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This  report  describes  a  research  sl(udy  conducted  by  Decisione  and  Designs,  Inc 
(DDI) ,  in  conjuotion  with  the  Army  Research  Institute  (ARI)  and  the  Army  Intel¬ 
ligence  Center  and  School  (USAICS) .  '^The  purpose  of  the^ study  was  to  test 
whether  tactical  Intelligence  analysis  could  be  improved  by  using  an  inter¬ 
active  Bayesian  inference  aid.  The  study  was  performed  with  experienced  tact¬ 
ical  intelligence  analysts  (officers  and  enlisted  personnel)  and  inexperienced 
but  trained  analysts  (officers  who  had  completed  their  training  at -USAICS). 

Two  hypotheses  guided  the  study.  The  first  hypothesis  was  that  experlencad 
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EXECUTIVE  SUMMARY 


The  subjective  revision  process  inherent  in  tactical 
intelligence  analysis  can  be  conceptualized  as  a  Bayesian 
inference  problem.  Experimental  research  in  which  partici¬ 
pants'  inferences  have  been  compared  with  those  prescribed  by 
Bayes'  Theorem,  for  tasks  other  than  tactical  intelligence 
analysis,  has  shown  that  people  have  considerable  difficulty 
making  judgments  consistent  with  Bayes'  Theorem.  Furthermore, 
the  research  suggests  that  this  difficulty  can  be  overcome  by 
using  interactive  computer  systems  that  rely  on  Bayes'  Theorem, 
rather  than  the  person,  to  integrate  large  amounts  of  data. 

A  research  study  was  conducted  to  test  whether  tactical 
intelligence  analysis  could  be  improved  by  using  an  interactive 
Bayesian  inference  aid.  It  was  found  that  it  could  be;  both 
experienced  and  inexperienced  {but  trained)  tactical  intelli¬ 
gence  analysts  demonstrated  significantly  improved  discrimina¬ 
tion  between  the  most  and  least  likely  opposing  force  (OPFOR) 
courses  of  action  as  determined  by  the  majority  of  analysts 
when  they  worked  with  an  interactive  Bayesian  inference  aid 
than  when  unaided.  Furthermore,  analysts  working  with  the  aid 
specified  a  final  rank  order  for  the  courses  of  action  that 
was  more  consistent  with  the  implications  of  the  intelligence 
data,  as  determined  by  the  assessed  likelihood  ratios  of  both 
aided  and  unaided  analysts  alike,  than  did  analysts  not  working 
with  the  aid. 

Future  research  efforts  should  focus  on  replicating  the 
results  reported  herein  with  more  complex  tactical  intelligence 
problems.  In  particular,  complexity  should  vary  in  three 
general  ways:  by  varying  characteristics  of  intelligence 
data,  by  linking  together  tactical  intelligence  analysis  with 
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subsequent  data  collection  and  tactical  decision  making,  and 
by  more  generally  representing  the  analysts'  working  environ¬ 
ment. 
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IMPROVING  TACTICAL  INTELLIGENCE  ANALYSIS: 
A  DEMONSTRATION 


1 . 0  INTRODUCTION 


Tactical  intelligence  analysis  necessarily  depends  on 
human  judgment,  for  analysts  must  continuously  revise  esti¬ 
mates  regarding  the  intent  of  opposing  forces  (OPFOR)  on  the 
basis  of  newly  collected  intelligence  data.  Adelman,  Donnell, 
and  Phelps  (1981)  used  Bayes'  Theorem  as  a  conceptual  frame¬ 
work  for  representing  the  subjective  revision  process  inher¬ 
ent  in  tactical  intelligence  analysis.  By  doing  so,  the 
large  body  of  psychological  research  on  Bayesian  inference 
was  brought  to  bear  on  tactical  intelligence  analysis.  This 
research  strongly  suggests  that  tactical  intelligence  analysis 
can  be  improved  by  using  Bayesian  inference  aids. 

Bayes'  Theorem  is  shown  in  equation  [1].  and  refer 
to  the  hypotheses  under  investigation,  which  in  tactical 
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intelligence  analysis  are  the  OPFOR  courses  of  action  (COAs) . 
D  refers  to  the  data  collected  prior  to  the  newly  collected 
intelligence  data,  which  is  represented  by  D'.  The  ratio  of 
prior  probabilities  P (H J  D) /P(H2l  D)  indicates  the  relative 
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likelihood  of  and  (the  possible  OPFOR  courses  of  action) 
prior  to  the  newly  collected  data.  The  ratio  of  conditional 
probabilities  P(D' 1  /P(D' |  H2»D) »  called  a  likelihood 

ratio,  indicates  the  extent  to  which  the  new  data  supports  each 
course  of  action,  for  it  represents  the  probability  of  observing 
that  datum  (or  indication)  if  the  OPFOR  were  actually  following 
a  given  course  of  action,  and  given  all  previous  data.  Finally, 
the  posterior  probabilities  P(Hj^|d’ ,D) /P(H2l  D' ,D)  indicate 
the  revised  likelihood  of  the  OPFOR  COAs  on  the  basis  of  all 
collected  intelligence  data.  As  represented  here,  Bayes* 

Theorem  represents  a  normative  rule  based  on  probability 
theory  for  indicating  how  tactical  intelligence  analysts 
should  revise  their  estimates  of  the  relative  likelihood  of 
different  OPFOR  courses  of  action  on  the  basis  of  all  collected 
intelligence  data. 

Fxtensive  reviews  of  experimental  research  in  which 
participants'  inferences  have  been  compared  with  those  prescribed 
by  Bayes*  Theorem,  for  tasks  other  than  tactical  intelligence 
analysis,  can  be  found  in  Fischer,  Edwards,  and  Kelly  (1978), 
Rapoport  and  Wallsten  (1972),  and  Slovic  and  Lichtenstein 
(1971).  This  research  strongly  suggests  that  when  given  new 
information,  analogous  to  collected  intelligence  data,  people 
have  considerable  difficulty  making  judgments  consistent  with 
Bayes*  Theorem.  In  general,  people  have  been  found  to  revise 
their  posterior  probabilities  as  to  the  likelihood  of  alterna¬ 
tive  hypotheses  in  the  same  direction  as  Bayes*  Theorem,  but 
they  do  not  revise  them  far  enough.  Such  "conservatism"  in 
judgment  occurs  because  people  extract  less  certainty  from  the 
data  than  they  should.  This  finding  could  have  great  implica¬ 
tions  for  tactical  intelligence  analysis,  which  is  a  sophisti¬ 
cated  Bayesian  inference  task.  If  tactical  intelligence 
analysts  are  conservative  information  processors,  then  they 
are  not  drawing  implications  from  the  data  as  fast  as  they 


could  be  with  Bayes'  Theorem.  Their  estimates  about  the 
relative  likelihood  of  different  OPFOP  courses  of  action  may 
well  be  suboptimal  because  they  will  not  have  sufficiently 
revised  their  opinions  to  take  full  account  of  the  certainty 
in  the  collected  data.  Consequently,  the  entire  analysis 
process  will  not  convey  as  much  information  to  friendly  comman 
ders  as  it  could,  thereby  reducing  their  time  for  tactical 
decision  planning  and  implementation. 

General  inference  aids,  called  Probabilistic  Information 
Processing  (PIP)  systems,  have  been  developed  to  ensure  that 
human  judgment  is  consistent  with  Bayes'  Theorem.  In  the 
formulation  proposed  by  Edwards,  Lindman,  and  Phillips  (1965), 
people  were  tasked  with  identifying  relevant  hypotheses, 
information  sources  that  could  discriminate  between  these 
hypotheses,  and  the  likelihood  ratios  linking  data  with  hypo¬ 
theses.  In  intelligence  analysis,  this  is  analogous  to  speci¬ 
fying  the  different  OPFOR  courses  of  action,  the  indicators 
related  to  each  course  of  action,  and  the  strength  of  the 
indicator-COA  relationship,  respectively.  The  task  of  aggre¬ 
gating  information  across  data  in  PIP  systems  was  assigned  to 
Bayes'  Theorem,  since  previous  research  had  indicated  that 
people  were  conservative  information  processors.  This  is 
analogous  to  using  Bayes'  Theorem  to  integrate  the  implication 
of  intelligence  data  used  to  make  the  overall  tactical  intelli 
gence  estimate.  Initial  efforts  by  Edwards,  Phillips,  Hays 
and  Goodman  (1968),  Kaplan  and  Newman  (1966),  and  VJheeler 
(1972)  all  found  PIP  systems  superior  to  unaided  inference; 

PIP  consistently  assigned  higher  posterior  probabilities  to 
the  true  hypotheses. 

The  research  re  cerenced  above  has  not  been  performed  with 
tactical  intelligence  analysts.  It  has  been  conducted  with 
college  students  under  controlled,  experimental  conditions 


using  relatively  simple  judgment  tasks  as  compared  to  those 
faced  by  tactical  intelligence  analysts.  Therefore,  one 
cannot  be  certain  that  experienced  intelligence  analysts  would 
exhibit  the  same  inconsistencies  and  limitations  in  judgment 
exhibited  in  the  cited  research.  There  is,  however,  reason  to 
suspect  that  they  will,  "Conservatism"  is  the  result  of 
people's  inability  to  intuitively  integrate  large  amounts  of 
uncertain  information  according  to  a  complex  normative  rule, 
Bayes'  Theorem,  Research  by  Tversky  and  Kahneman  (1974)  has 
shown  that  people  use  simpler  heuristics  instead  of  Bayes' 
Theorem  to  integrate  the  data.  There  appears  to  be  no  reason 
why  intelligence  analysts  should  be  able  to  perform  this 
complex  cognitive  task  any  better  than  college  students,  for 
they  receive  no  training  in  using  Bayes'  Theorem  to  integrate 
intelligence  data.  Conservatism  on  the  part  of  tactical 
intelligence  analysts  could  well  result  in  suboptimal  esti¬ 
mates  of  the  relative  likelihood  of  possible  OPFOR  courses  of 
action,  thereby  reducing  the  time  available  for  tactical 
decision  making  and  implementation  by  friendly  commanders. 

The  research  cited  above  suggests  that  conservatism  can  be 
reduced  by  using  PIP  systems  that  rely  on  Bayes'  Theorem., 
rather  than  the  person,  to  integrate  large  amounts  of  data. 

This  report  describes  a  research  study  conducted  by 
Decisions  and  Designs,  Inc.  (DDI) ,  in  conjunction  with  the 
Army  Research  Institute  (ARI)  and  the  U.S.  Army  Intelligence 
Center  and  School  (USAICS) .  The  purpose  of  the  study  was  to 
test  whether  tactical  intelligence  analysis  could  be  improved 
by  using  an  interactive  Bayesian  inference  aid.  The  study  was 
performed  with  experienced  tactical  intelligence  analysts 
(officers  and  enlisted  personnel)  and  inexperienced  but  trained 
analysts  (officers  who  had  completed  their  training  at  USAICS), 
Two  hypotheses  guided  the  study.  The  first  hypothesis  was  that 
experienced  and  inexperienced  analysts  working  with  the  aid 


would  be  better  able  to  discriminate  between  the  most  and  least 
likely  OPFOR  courses  of  action  than  analysts  working  without  the 
aid.  The  second  hypothesis  was  that  this  improved  performance 
would  be  the  result  of  significantly  less  conservatism  with  than 
without  the  interactive  Bayesian  aid. 

TO  test  these  hypotheses,  the  analyses  focused  on  the 
extent  to  which  the  relative  likelihood  estimates  for  the 
OPFOR  COAs  changed  after  receiving  intelligence  data  favoring 
two  of  the  four  COAs  in  the  representative  tactical  intelligence 
problem  used  in  the  study.  The  first  hypothesis  would  be  con¬ 
firmed  by  finding  that  aided  analysts,  both  experienced  and 
inexperienced  alike,  had  significantly  higher  posterior  proba¬ 
bilities  than  unaided  analysts  for  the  OPFOR  courses  of  action 
considered  most  likely  by  the  majority  of  analysts.  The 
second  hypothesis  would  be  confirmed  by  finding  that  aided 
analysts  had  a  significantly  smaller  difference  between  their 
theoretical  (i.e.,  Bayesian)  and  actual  (i.e.,  assessed) 
posterior  probabilities  than  did  unaided  analysts.  Taken 
together,  these  findings  would  strongly  suggest  that  tactical 
intelligence  analysts  are  significantly  less  conservative  and, 
therefore,  significantly  better  able  to  differentiate  between 
the  more  and  less  likely  OPFOR  courses  of  action  when  they  can 
use  an  interactive  PIP  system  that  employs  Bayes'  Theorem  to 
integrate  uncertain  data.  Discussions  with  experienced  tacti¬ 
cal  intelligence  analysts  indicated  that  such  a  finding  would 
have  significant  implications  for  improving  tactical  intelligence 
collection  and  analysis. 

The  remaining  sections  of  this  report  detail  the  various 
elements  of  the  research  study.  Section  2.0  describes  the 
technical  approach  used  in  implementing  the  study.  Section  3.0 
presents  the  results  of  the  analyses  explicitly  designed  to  test 
the  two  hypotheses  guiding  the  study.  Section  4.0  presents 


the  results  of  analyses  designed  to  test  whether  aided  and  unaided 
analysts  differed  on  other  measures.  Section  5.0  presents  data 
on  how  individual  analysts  used  the  aid  to  revise  their  opinion 
of  OPPOR  intent.  Finally,  Section  6.0  presents  the  conclusions 
and  recommendations  for  future  research. 


2.0  TECHNICAL  APPROACH 


This  section  describes,  In  turn,  <1)  the  general  design 
of  the  study,  (2)  the  general  background  of  the  tactical 
intelligence  analysts  who  participated  in  the  study,  (3)  the 
tactical  intelligence  exercise  they  worked  on,  (4)  the  inter¬ 
active  Bayesian  inference  aid  (or  PIP  system)  that  some  of  the 
participants  used  when  working  the  problem,  (5)  the  different 
procedures  followed  by  participants  in  the  different  research 
conditions,  and  (6)  the  statistical  analysis  used  to  test  the 
hypotheses  guiding  the  study. 

2 . 1  General  Design 

Two  factors  were  manipulated  in  performing  the  research 
reported  herein:  (1)  whether  or  not  the  participants  used  an 
interactive  Bayesian  inference  aid  when  working  the  problem, 
and  (2)  whether  or  not  the  participants  were  experienced 
tactical  intelligence  analysts.  The  factors  were  crossed  with 
each  other,  resulting  in  the  following  four  conditions: 
experienced  aided,  experienced  unaided,  inexperienced  aided, 
and  inexperienced  unaided.  Each  participant  estimated  the 
relative  likelihood  of  the  four  possible  OPFOR  courses  of 
action  after  reading  the  scenario,  but  prior  to  receiving 
intelligence  data  (i.e.,  messages),  and  again  after  receiving 
all  the  data.  The  study  focused  on  the  extent  to  which,  under 
each  of  the  four  conditions,  the  relative  likelihood  estimates 
for  the  OPFOR  courses  of  action  changed  after  participants 
received  the  data. 

It  is  important  to  point  out  that,  in  practice,  tactical 
intelligence  analysts  are  usually  not  required  to  estimate  the 
relative  likelihoods  of  the  different  OPFOR  courses  of  action, 
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although  on  occasion  they  may  be  asked  to  do  so  by  the  friendly 
commander.  What  they  are  required  to  do,  either  verbally  or 
through  routinely  submitted  summary  reports,  is  to  rank  order 
the  possible  OPPOR  courses  of  action  and  to  keep  the  commander 
informed  of  significant  OPPOR  activity.  The  study  took  into 
account  these  operational  requirements  in  the  following  two 
ways:  first,  by  having  all  participants  answer  three  behav¬ 
ioral  questions  after  each  message  and,  second,  by  having  them 
write,  after  the  final  message,  a  summary  report  indicating 
the  rank  order  of  the  OPPOR  courses  of  action  (from  most  to 
least  likely)  and  the  reasons  for  their  position.  It  is,  of 
course,  quite  possible  for  tactical  intelligence  analysts  to 
be  more  or  less  conservative  in  their  subjective  estimates 
about  the  relative  likelihood  of  different  OPPOR  courses  of 
action  without  that  variance  affecting  (1)  the  final  rank 
order  given  to  the  OPPOR  courses  of  action,  (2)  the  quality  of 
their  summary  report,  or  (3)  their  behaviors  after  each  message. 


Participants 


There  were  two  groups  of  research  participants,  those  who 
were  experienced  tactical  intelligence  analysts  and  those  who 
were  not.  Twelve  experienced  analysts  participated^  three 
were  instructors  at  USAICS  and  nine  were  practicing  analysts 
in  the  Washington,  D.C.  area,  either  at  the  Intelligence  and 
Security  Command  (INSCOM)  or  the  Intelligence  Threat  and 
Analysis  Center  (ITAC).  All  eighteen  inexperienced  analysts 
were  Army  officers  who  had  just  completed  training  at  USAICS. 


2 . 3  Research  Exercise 

All  research  participants  worked  through  an  abridged 
version  of  a  training  exercise  developed  by  the  U.S.  Army 
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Comniancl  and  General  Staff  College.  The  exercise  is  called 
"Operation  Jayhawk;  Control  and  Coordination  of  Division 
Operations."  The  participants*  task  during  the  research  study 
was  to  estimate  the  relative  likelihood  of  four  OPPOR  avenues 
of  approach  for  attacking  U.S.  forces.  This  estimate  was  to 
be  baaed  on  (1)  background  information  about  U.S.  and  OPPOR 
force  composition  and  disposition,  as  well  as  terrain  and 
weather  considerations,  and  {2)  ten  messages  representing 
intelligence  data  about  OPPOR  activity  collected  sequentially 
over  a  two-day  period.  The  messages  varied  in  length  and 
diagnosticity,  i.e.,  the  extent  to  which  they  supported  one 
OPPOR  course  of  action  over  another,  but  they  generally  sup¬ 
ported  COAs  #?.  and  #3.  Each  participant  had  a  map  of  the  area 
under  consideration  and  a  transparent  acetate  overlay  so  that 
force  composition  and  disposition,  key  military  terrain  features, 
and  reported  OPPOR  activity  could  be  recorded.  (The  actual 
background  scenario  and  messages  can  be  found  in  Appendix  A. ) 


2.4  The  Bayesian  Inference  Aid 


A  simple  Bayesian  inference  aid  (or  PIP  system)  was 
developed  by  the  research  team  and  DDI  computer  scientists 
and  implemented  on  an  IBM  5110/20  portable  computer.  Dis¬ 
cussions  and  pretesting  prior  to  the  study  clearly  indicated 
that  tactical  intelligence  analysts  would  never  use  the  aid  in 
practice,  no  matter  what  the  research  findings,  if  they  could 
not  revise  the  posterior  probability  estimates  generated  by 
it.  Consequently,  the  aid  was  designed  to  be  a  highly  inter¬ 
active  one  that  permitted  the  user  to  modify  the  aid's  output 
and  to  see  the  implications  of  those  modifications. 


The  aid  is  implemented  through  a  four-step  procedure.  (To 
assist  the  reader  in  understanding  the  description  that  follows, 
Appendix  B  presents  computer  printouts  showing  how  one  of  the 
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experienced  analysts  used  the  interactive  Bayesian  aid  to 
complete  this  four-step  sequence.) 

(1)  The  tactical  intelligence  analyst  first  defines  the 
number  (n)  (in  this  case,  n  ■  4)  OPPOR  courses  of  action 
under  consideration. 

(2)  The  analyst  then  enters  the  relative  li)celihood  of 
the  COAs  on  the  basis  of  all  prior  information.  This  is 
accomplished  in  two  steps.  First,  the  analyst  rank  orders  the 
COAs  from  most  to  least  IDcely;  second,  the  analyst  specifies 
the  prior  odds,  that  is,  how  many  times  as  likely  the  most 
likely  COA  is  than  each  of  the  other  COAs.  The  aid  then 
computes  the  prior  probabilities  for  the  COAs  such  that  they 
sum  to  1.0.  The  analyst  can  either  accept  or  change  these 
probabilities.  If  the  prior  probabilities  are  changed,  the 
aid  computes  the  prior  odds  that  the  analyst  would  have  had 

to  have  given  to  result  in  the  specified  probabilities.  The 
analyst  can  then  modify  these  new  prior  odds  and  see  the 
implied  prior  probabilities,  modify  the  probabilities  directly 
(the  aid  ensures  that  they  sum  to  1,0)  and  see  the  implied 
prior  odds,  or  revise  neither  the  prior  odds  or  probabilities. 
This  step  ends  when  the  analyst  accepts  a  set  of  prior  proba¬ 
bilities;  these  are  the  only  prior  probabilities  and  odds  that 
the  aid  uses  and  stores  for  subsequent  consideration. 

(3)  For  a  given  datum,  the  analyst  enters  a  brief  title 
summarizing  the  message.  The  analyst  then  rank  orders  the 
COAs  on  the  basis  of  how  likely  one  is  to  see  that  datum  for 
each  COA.  The  COA  ranked  first  is  the  COA  for  which  one  would 
be  most  likely  to  see  the  datum;  the  COA  ranked  last  is  the 
COA  for  which  one  would  be  least  likely  to  see  that  datum. 
After  rank-ordering  the  COAs  in  terms  of  the  likelihood  of  the 
datum,  the  analyst  then  specifies  how  many  times  as  likely  one 


is  to  see  the  datum  for  the  most  likely  COA  as  compared  to 
each  of  the  other  COAs. 

These  numbers  represent  the  likelihood  ratios  in  Bayes* 
Theorem,  Examination  of  equation  tH  in  the  introduction  to 
this  report  shows  that  they  represent  a  ratio  of  two  conditional 
probabilities,  that  I:',  P(D' I  /P(D' |  .  This  ratio 

indicates  the  relative  likelihood  of  receiving  the  datum  |D') 
given  that  the  OPFOR  was  actually  pursuing  C0A#1  (Hj^)  versus 
C0A#2  (H2)»  and  taking  into  account  all  previous  data  (D).  If 
the  ratio  is  greater  than  1,0,  then  given  all  previous  data  it 
is  more  likely  to  observe  the  datum  if  the  OPFOR  is  pursuing 
COA#l  than  if  they  are  pursuing  COA#2,  If  the  ratio  is  less 
than  1.0,  then  one  is  more  likely  to  receive  the  datum  if  the 
OPFOR  is  actually  pursuing  COA#2.  If  the  ratio  equals  1,0, 
then  the  datum  is  just  as  likely  to  have  been  generated  by 
C0A#1  or  COAi2.  Since  the  aid  asks  the  analysts  first  to  rank 
order  the  COAs  on  the  basis  of  how  likely  they  are  to  generate 
the  datum  D',  and  then  to  make  the  ratio  comparisons,  the  ratios 
have  to  be  greater  than  or  equal  to  1,0.  The  larger  the 
likelihood  ratio  between  two  COAs,  the  more  diagnostic  the 
datum  for  differentiating  between  COAs.  In  tactical  intel¬ 
ligence  analysis  terms,  the  higher  the  likelihood  ratio,  the 
better  the  indication  of  OPFOR  intent. 

(4)  Once  the  analyst  specifies  the  likelihood  ratios, 
the  aid  uses  Bayes*  Theorem  to  compute  the  posterior  odds  and 
probabilities  for  the  OPFOR  COAs.  It  is  important  to  point 
out  to  readers  who  are  not  familiar  with  Bayes*  Theorem  that 
the  posterior  odds  and  likelihood  ratios  are  distinctly  dif¬ 
ferent  terms  in  Bayes*  Theorem.  The  posterior  odds  indicate 
the  relative  likelihood  of  two  COAs,  given  all  the  data;  the 
likelihood  ratio  indicates  the  relative  likelihood  of  the 
datum  (D*),  given  the  two  COAs  and  all  previous  data  (D).  It 


is  quite  possible  to  have  a  high  likelihood  ratio  implying  that 
a  datum  was  probably  generated  by  an  OPFOR  COA  that,  given  all 
the  data,  would  probably  not  be  followed  by  the  OPFOR  (i.e,, 
have  a  low  posterior  probability) >  this  might  occur,  for  example, 
if  the  OPFOR  was  trying  to  deceive  the  friendly  forces. 

It  is  important  to  point  out  to  readers  familiar  with 
Bayes'  Theorem  that  pretesting  discovered  that  tactical  intel¬ 
ligence  data  is  conditionally  dependent;  the  probability  of 
datum  given  a  particular  COA  depends  on  the  set  of  previously 
received  data.  Consequently,  participants  were  told  to  specify 
likelihood  ratios  baaed  on  conditional  dependence,  not  condi¬ 
tional  independence,  which  was  so  often  done  in  previous 
research. 

The  analyst  can  either  accept  or  change  the  posterior 
probabilities  generated  by  the  aid.  If  the  posterior  proba¬ 
bilities  are  changed,  the  aid  then  computes  the  likelihood 
ratios  that,  in  conjunction  with  the  prior  odds,  would  have 
generated  the  newly  specified  posterior  probabilities.  The 
aid  also  displays  the  original  likelihood  ratios  so  that  the 
analyst  can  compare  the  likelihood  ratios  implied  by  the  newly 
specified  posterior  probabilities  with  those  assessed  originally. 
The  analyst  now  has  the  opportunity  to  change  the  likelihood 
ratios,  thereby  generating  new  posterior  probabilities.  This 
iteration  continues  until  the  analyst  is  pleased  with  the 
likelihood  ratios  and  posterior  probabilities  for  each  datum. 

The  aid  uses  and  stores  only  the  last  set  of  posterior  proba¬ 
bilities  and  likelihood  ratios.  When  a  new  datum  is  received, 
these  posterior  probabilities  now  become  the  prior  probabili¬ 
ties  in  Bayes'  Theorem,  and  the  analyst  need  only  generate 
likelihood  ratios  for  the  new  datum  in  order  to  revise  the 
estimate  of  the  relative  likelihood  (i.e.,  the  posterior 
probabilities)  of  the  OPFOR  COAs. 
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2.5  Procedures 


i 

The  procedures  followed  by  the  participating  analysts 
depended  on  whether  the  analysts  worked  with  the  Baysian 
inference  aid  or  not.  Analysts  working  with  the  aid  first 
I  read  the  background  scenario  describing  the  Jayhawk  exercise. 

After  doing  so,  they  used  the  interactive  Bayesian  inference 
aid  to  help  them  estimate  the  prior  odds  and  probabilities  for 
the  four  OPFOR  COAs  (called  Avenues  of  Approach  in  the  Jayhawk 
I  exercise).  Then,  the  analysts  received,  in  sequential  fashion, 

separate  messages  describing  observed  OPFOR  activity.  For 
each  message,  the  analysts  estimated  the  likelihood  ratios  in 
the  manner  described  in  Section  2.4,  and  the  aid  calculated 
I  the  posterior  probabilities  and  odds.  The  analysts  used  the 

aid's  interactive  capabilities  until  they  were  satisfied  with 
the  posterior  probabilities.  Then,  they  received  the  next 
message  and  repeated  the  above  procedure.  Eight  inexperienced 
4  analysts  and  seven  experienced  analysts  participated  in  the 

aided  condition;  of  the  seven  experienced  analysts,  three  were 
instructors  at  USAISC  and  four  were  practicing  tactical  intelli¬ 
gence  analysts  from  agencies  in  the  Washington,  D.C.  area. 

The  procedures  followed  in  the  unaided  condition  depended 
on  whether  the  participant  was  an  inexperienced  or  an  experienced 
analyst.  The  inexperienced  analysts  in  the  unaided  condition 
t  followed  the  same  procedures  as  the  analysts  in  the  aided 

condition,  but  they  did  not  use  the  Bayesian  inference  aid  to 
help  them  estimate  the  prior  odds,  the  likelihood  ratios,  or 
the  posterior  odds.  The  experienced  analysts  in  the  unaided 
»  condition  just  estimated  the  prior  odds  and  the  posterior  odds 

after  all  ten  messages;  they  did  not  estimate  the  likelihood 
ratios  and  posterior  odds  for  each  message.  The  procedures  in 
the  unaided  condition  were  modified  for  the  experienced  analysts 
»  for  three  reasons;  (1)  few  practicing  tactical  intelligence 
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analysts  in  the  Washington,  D.C.  area  were  available  for  parti¬ 
cipation  during  the  time  of  the  study;  (2)  there  were  time  con¬ 
straints  when  working  with  them;  and  (3)  future  implementation 
concerns  required  that  time  be  given  to  letting  the  analysts 
gain  experience  with  the  Bayesian  inference  aid  after  participa 
ting  in  the  study.  Ten  inexperienced  analysts  and  five  experi¬ 
enced  analysts  participated  in  the  unaided  condition.  (Copies 
of  the  forms  used  to  estimate  the  prior  odds,  likelihood  ratios 
and  posterior  odds  can  be  found  in  Appendix  C. 

All  participants  did  two  other  activities  in  addition  to 
making  the  probability  estimates  described  above.  First,  they 
answered  the  following  three  behavioral  questions  after  each 
message:  (1)  V7ould  they  request  an  immediate  briefing  with 
the  Division  Commander?  (2)  Would  they  request  additional 
information  from  Corps?  (3)  Could  they  conclude  that  the 
OPFOR  had  selected  a  primary  avenue  of  approach?  Second, 
after  receiving  all  the  messages  and  estimating  their  final 
posterior  probabilities,  the  analysts  wrote  a  summary  report 
rank  ordering  the  COAs  from  most  to  least  likely  and  justify¬ 
ing  their  position.  These  two  additional  activities  were 
Included  in  the  study  in  an  effort  to  determine  whether  the 
behavior  of  tactical  intelligence  analysts  (or  more  accurately, 
experimental  efforts  to  model  that  behavior)  either  depended 
on  the  level  of  analyst's  experience  or  was  affected  by  working 
with  the  aid.  (A  copy  of  the  questionnaire  and  summary  report 
form  can  be  found  in  Appendix  D.) 

Experienced  and  inexperienced  analysts  in  the  aided  and 
unaided  conditions  received  a  tutorial  on  Bayesian  inference. 
Participants  in  the  aided  condition  also  received  instruction 
on  using  the  aid.  The  researchers  monitored  the  subjects' 
behavior  for  the  first  few  messages  to  ensure  that  the  partici¬ 
pants  could  use  the  aid  correctly.  In  general,  efforts  were 
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made  to  ensure  that  participants  understood  their  task,  felt 
free  to  ask  questions,  and  had  sufficient  time  to  complete  their 
work. 


2.6  Statistical  Analysis 

Analysis  of  Variance  (ANOVA)  was  used  to  test  statistically 
whether  the  four  conditions  (aided-experienced ,  aided-inexperi- 
enced,  unaided-experienced,  and  unaided-inexperienced)  differed, 
in  general,  in  their  estimates  of  the  relative  likelihood  of  the 
four  OPFOR  COAs.  The  probabilities  for  the  four  COAs  could  not 
be  used,  however,  because  they  were  not  independent,  as  required 
for  an  AKOVA;  one  can  calculate  the  fourth  probability  from  the 
first  three  since  the  four  probabilities  must  sum  to  1.0. 
Consequently,  odds  estimates  for  three  COA  comparisons  were  used 
instead  of  probabilities  to  ensure  independence  in  the  ANOVA,  In 
addition,  the  logarithm  of  each  participant's  odds  estimate  for 
each  of  the  three  COA  comparisons  was  calculated  to  ensure  that 
the  dependent  variables  were  additive  (probabilities  are  multi¬ 
plicative),  as  required  by  an  ANOVA. 

The  following  three  independent  COA  comparisons  were  used 
in  the  ANOVA:  avenues  of  approach  #2  vs  #1,  #3  vs  #2,  and  #3 
V8  #4.  These  COA  comparisons  were  chosen  because  preliminary 
examination  of  the  data  clearly  indicated  that  #2  and  #3  were 
the  two  most  likely  OPFOR  COAs  and  that  #1  and  #4  were  the  two 
least  likely  COAs,  in  the  opinion  of  the  majority  of  the  ana¬ 
lysts  after  they  had  received  all  ten  messages.  The  selected  , 
COA  comparisons  represent  the  smallest  number  of  independent 
comparisons  that  include  all  four  COAs  and,  at  the  same  time, 
permit  one  to  evaluate  group  differences  between  (a)  the  two 
most  likely  OPFOR  COAs  and  (b)  each  of  the  most  likely  COAs 
with  at  least  one  of  the  least  likely  COAs. 


15 


tULAiuiAa 


lUkteMHIIliUii 


In  summary,  AKOVA  on  the  logarithms  of  the  odds  estimates 
for  COA  comparisons  #2  vs  #1,  #3  vs  #2,  and  #3  vs  #4  were  used 
to  test  the  hypotheses  guiding  the  research  study.  The  type 
of  ANOVA  design  depended  on  the  dependent  measure  (e.g.,  the 
posterior  odds  or  the  mean  rank  orders)  being  analyzed.  The 
particular  ANOVA  design  used  in  the  analysis  is  described 
prior  to  presenting  the  results  for  that  dependent  measure. 


3.0  RESULTS:  CONFIRMATION 
OF  GUIDING  HYPOTHESES 


Two  hypotheses  guided  the  study.  The  first  hypothesis 
was  that  experienced  and  inexperienced  analysts  working  with 
the  aid  would  be  better  able  to  discriminate  between  the  most 
and  least  likely  OPFOR  courses  of  action  than  analysts  working 
without  the  aid.  The  second  hypothesis  was  that  this  improved 
performance  would  be  the  result  of  significantly  less  conserva¬ 
tism  with  than  without  the  interactive  Bayesian  aid.  To  test 
these  hypotheses,  the  analyses  focused  on  the  extent  to  which 
the  relative  likelihood  estimates  for  the  OPFOR  COAs  changed 
after  receiving  intelligence  data  favoring  two  of  the  four 
COAs  in  the  representative  tactical  intelligence  problem  used 
in  the  study.  This  section  presents  the  results  of  these 
analyses  with  respect  to  each  of  the  two  hypotheses. 

3.1  Hypothesis  I;  Improved  Discrimination  with  the  Aid 

The  following  results  would  support  the  first  hypothesis, 
that  experienced  and  inexperienced  analysts  are  better  able  to 
discriminate  between  the  most  and  least  likely  OPFOR  COAs  with 
than  without  the  aid;  (1)  there  were  no  statistically  signi¬ 
ficant  differences  in  the  logarithms  for  the  groups'  final 
theoretical  posterior  odds  for  the  three  COA  comparisons, 
indicating  that  the  groups  drew  the  same  implicatio»^8  from  the 
data  and  had  similar  likelihood  ratios  for  the  ten  messages; 

(2)  there  were  statistically  significant  differences  in  the 
logarithms  for  the  groups'  final  assessed  posterior  odds,  with 
the  aided  groups  having  significantly  larger  logarithms  than  the 
unaided  groups  for  the  COA  comparisons  (particularly  for 
comparisons  #2  vs  #1  and  for  #3  vs  #4),  indicating  that  aided 
analysts  were  better  able  to  use  the  intelligence  data  to 
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discriminate  between  the  most  and  least  likely  OPFOR  COAs;  and 
(3)  there  was  a  significantly  larger  difference  between  the 
logarithms  of  the  assessed  posterior  and  prior  odds  for  the 
aided  than  unaided  groups,  indicating  that  aided  analysts  were 
better  able  to  use  the  intelligence  data  to  move  away  from 
their  initial  position. 

It  is  important  to  note  that  confirming  the  guiding 
hypothesis  does  not  necessarily  imply  that  participants  using 
the  aid  were  more  accurate  than  participants  not  using  the 
aid.  Unfortunately,  the  Jayhawk  exercise  did  not  indicate 
what  the  relative  likelihood  of  the  four  OPFOR  COAs  should  be 
prior  to  or  after  receipt  of  the  intelligence  data;  it  neither 
rank  ordered  the  COAs  nor  provided  prior  or  posterior  proba¬ 
bilities.  Since  there  was  no  external  criterion  in  the  Jayhawk 
exercise,  accuracy  could  not  be  measured  in  the  research 
study. 

It  was,  however,  possible  to  measure  the  general  level  of 
agreement  among  the  participants.  Presumably,  the  analysts 
.should  agree,  in  general,  on  the  most  and  least  likely  OPFOR 
COAs  and  on  the  likelihood  ratios,  regardless  of  whether  or 
not  they  use  the  aid.  What  is  hypothesized  to  differ  is  the 
posterior  probabilities;  with  the  aid,  they  should  be  much 
higher  for  the  COAs  that,  in  general,  the  analysts  consider 
most  likely  and,  conversely,  much  lower  for  the  COAs  that  the 
analysts,  in  general,  consider  least  likely.  Such  a  finding 
would  suggest  that  tactical  intelligence  analysts  are  better 
able  to  differentiate  between  OPFOR  courses  of  action  when 
they  can  use  a  PIP  system  that  utilizes  Bayes'  Theorem  to 
integrate  uncertain  data. 

3.1.1  Theoretical  posterior  odds  -  If  the  aided- 
experienced,  aided-inexperienced ,  and  unaided-inexperienced 
groups  drew  the  same  implications  from  the  tactical  intelli- 
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gence  data  in  the  ten  measages,  then  their  assessed  likelihood 
ratios  should  have  been  sufficiently  similar  to  result  in 
similar  theoretical  posterior  odds  for  the  three  COA  comparisons, 
A  one-between-subjeets  and  one-within-subjects  ANOVA  design  was 
used  to  test  whether  the  final  theoretical  posterior  odds 
(i.e.,  those  calculated  from  the  analysts'  initially  assessed 
likelihood  ratios  using  Bayes*  Theorem)  differed  for  these 
three  groups.  The  unaided-experienced  group  did  not  estimate 
likelihood  ratios,  so  one  cannot  calculate  their  theoretical 
posterior  odds.  The  three  groups  represented  levels  on  the 
between-subjects  factor;  the  three  COAs  represented  levels  on 
the  within-subjecta  factor.  The  dependent  measure  was  the 
logarithms  of  the  theoretical  posterior  odds  for  the  three  COA 
comparisons. 

Figure  3-1  presents,  for  each  group,  the  mean 
logarithm  of  the  theoretical  posterior  odds  for  each  of  the 
three  COA  comparisons,  (Note;  Throughout  the  presentation, 

COA  and  AOA  are  used  interchangeably  because  the  Jayhawk  Exer¬ 
cise  refers  to  the  courses  of  action  as  avenues  of  approach.) 

The  larger  the  logarithm,  the  more  likely  the  first  COA  in  the 
pair.  The  logarithms  of  the  theoretical  posterior  odds  indicate 
that,  on  the  basis  of  the  assessed  likelihood  ratios  for  the 
ten  messages,  all  three  groups  considered  avenue  of  approach  #2 
more  likely  than  #1,  #3  more  likely  than  12,  and  #3  more  likely 
than  #4. 

Table  3-1  shows  the  ANOVA  results  based  on  these 
data.  The  only  significant  effect  was  for  avenues  of  approach 
(AOA);  the  logarithms  of  the  theoretical  posterior  odds  for  #3 
vs  #2  were  significantly  smaller  than  those  for  #2  vs  #1  or  for 
#3  vs  #4,  indicating  that  the  analysts  thought  the  messages 
supported  avenues  of  approach  #3  and  #2  over  #1  and  #4.  There 
were  no  significant  group  differences.  The  failure  to  find  a 
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Source  of  Variation 
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Subject/Group 
Within  Subjects 
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101.479  22  4.613 
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significant  group  effect  is  consistent  with  the  proposition 
that  the  three  groups  drew  similar  implications  from  the  data 
(i.e.,  they  had  similar  likelihood  ratios  for  the  ten  messages), 
and  that  the  data  indicated  that  avenues  of  approach  #3  and  (to 
a  lesser  extent)  #2  were  significantly  more  likely  than  avenues 
of  approach  #1  and  #4. 

3,1.2  Assessed  posterior  odds  -  If  the  aid  improved  the 
performance  of  tactical  intelligence  analysts  regardless  of 
their  level  of  experience,  then  the  aided-experlenced  and 
alded-inexperlenced  groups  should  have  similar  and  signifi¬ 
cantly  larger  logarithms  for  the  COA  comparisons  than  would 
the  unaided-experienced  and  unaided-inexperienced  groups. 
Furthermore,  based  on  the  results  for  the  theoretical  posterior 
odds,  this  effect  should  be  most  pronounced  for  COA  comparisons 
#2  vs  #1  and  for  #3  vs  #4,  since  the  likelihood  ratios  assessed 
by  the  analysts  themselves  indicated  that  the  data  strongly 
supported  avenues  of  approach  #3  and  #2  over  avenues  of  approach 
#1  and  #4. 


A  two-between- subjects  and  one-within-subjects  ANOVA 
design  was  used  to  test  whether  the  final  assessed  posterior 
odds  differed  for  the  four  groups.  The  two-between-subjects 
factors  were  the  level  of  the  analyst's  experience  and  whether 
or  not  the  aid  was  used,  since  each  participant  was  in  only 
one  of  the  study's  four  groups.  The  COA  factor  was  with in¬ 
subjects  because  the  three  COA  comparisons  were  made  for  each 
participant.  The  dependent  measure  was  the  logarithms  of  the 
final  assessed  posterior  odds  for  the  three  COA  comparisons. 

Figure  3-2  presents,  for  each  group,  the  mean 
logarithm  of  the  final  assessed  posterior  odds  for  each  of  the 
three  COA  comparisons.  The  larger  the  logarithm,  the  more 
likely  the  first  avenue  of  approach  in  the  pair.  A  logarithm 
of  zero  indicates  that  the  two  avenues  of  approach  were  equally 
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likely;  a  negative  logarithm  indicates  that  the  second  avenue 
of  approach  in  the  pair  was  the  more  likely  one.  The  mean 
logarithms  for  the  COA  comparisons  made  by  all  four  groups  were 
positive.  Comparing  Figures  3-1  and  3-2  indicates  that  each 
group's  final  assessed  posterior  odds  were  in  the  same  direction, 
although  not  as  extreme,  as  their  theoretrical  posterior  odds. 

Table  3-2  shows  the  ANOVA  results  based  on  these 
data.  There  was  a  significant  main  effect  for  Aid  and  for  AOA; 
there  was  also  a  significant  Aid-by-AOA  interaction.  The  main 
effect  for  Aid  indicates  that  there  were  significant  differences 
in  the  mean  logarithms  of  the  groups'  final  assessed  posterior 
odds  for  the  COA  comparisons.  Figure  3-2  shows  that  the  two 
aided  groups  had  similar  and  larger  logarithms  than  the  two  un¬ 
aided  groups  for  all  three  COA  comparisons,  as  hypothesized. 

The  Aid-by-AOA  interaction  indicates  that  the  size  of  this  group 
difference  depended  on  the  COA  comparison;  the  group  differences 
were  most  pronounced  for  COA  comparisons  #2  vs  #1  and  for  #3  vs 
#4.  Finally,  the  AOA  main  effect  occurred  because  the  logarithms 
of  the  final  assessed  posterior  odds  for  #3  vs  #2  were  significantly 
smaller  than  those  for  #2  vs  #1  and  for  #3  vs  #4,  for  all  four 
groups.  These  last  two  findings  were  consistent  with  the  results 
for  the  theoretical  posterior  odds,  which  indicated  that  the 
likelihood  ratios  based  on  the  messages  strongly  supported  avenues 
of  approach  #3  and  #2  over  avenues  of  approach  #1  and  #4. 

In  sum,  the  results  reported  in  this  section 
confirm  the  first  hypothesis,  which  was  that  experienced  and 
inexperienced  analysts  alike  would  be  better  able  to  discrimi¬ 
nate  between  the  most  and  least  likely  OPFOR  COAs  when  they 
worked  with  an  interactive  Bayesian  inference  aid.  The  next 
section  will  show  that  this  improved  performance  resulted 
because  aided  analysts  were  better  able  than  unaided  analysts 
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SOURCE  OP  VARIATION 


Between  Subiects 


Aid  (A) 

41.312 

1 

41.312 

19.899** 

Experience  (E) 

.074 

1 

.074 

.036 

Aid  X  Experience  (AE) 

.550 

1 

.550 

.265 

> 

Subject/AE 

53.986 

26 

2.076 

v;lthin  Subjects 

AOA 

48.569 

2 

24.285 

20.054** 

1 

Aid  X  AOA 

11.576 

2 

5.788 

4.780* 

Experience  x  AOA 

3.132 

2 

1.566 

1.293 

A  X  E  X  AOA 

1.469 

2 

.735 

.607 

Subjects  X  AOA/AE 

62.996 

52 

1.211 

*  ■  p  <  .05 

**  »  p  <  ,001 
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25 


to  use  the  implications  of  the  messages  to  move  away  from 
their  initial  position. 

3.1.3  Difference  between  asBessed  posterior  and  prior 
odds  -  If  the  aid  helped  experienced  and  inexperienced  analysts 
alike  to  use  the  intelligence  data  to  move  away  from  their 
initial  position,  then  the  difference  in  the  logarithms  of  the 
posterior  and  prior  odds  for  the  aided-experienced  and  aided- 
inexperienced  groups  should  be  similar  and  significantly 
larger  than  the  differences  for  the  unaided-experienced  and 
unaided-inexperienced  groups.  A  two-between-subjects  and  one- 
within-subjects  ANOVA  design  was  used  to  test  this  proposition. 
The  two  between-subjects  factors  were  the  level  of  the  analyst's 
experience  and  whether  or  not  the  aid  was  used;  the  one  within- 
subjects  factor  was  the  COA  comparisons.  The  dependent  measure 
was  the  difference  in  the  logarithms  for  the  assessed  posterior 
and  prior  odds  for  the  three  COA  comparisons. 

Figure  3-3  presents,  for  each  group,  the  mean 
difference  in  the  logarithms  for  the  assessed  posterior  and 
prior  odds  for  the  three  COA  comparisons.  Positive  differences 
indicate  that  the  posterior  odds  were  larger  than  the  prior 
odds;  negative  differences  indicate  that  the  prior  odds  were 
larger  than  the  posterior  odds.  Table  3-3  shows, the  ANOVA 
results  based  on  these  data.  Significant  main  effects  were 
found  for  the  Aid  and  for  AOA;  a  significant  Aid-by-AOA  inter¬ 
action  was  found  too.  The  Aid  main  effect  indicates  that  the 
mean  differences  in  the  logarithms  of  the  posterior  and  prior 
odds  for  the  aided-experienced  and  aided-inexperienced  groups 
were  similar  and  significantly  larger  than  the  differences  for 
the  two  unaided  groups  over  all  three  COA  comparisons.  The 
Ald-by-AOA  Interaction  indicates  that  the  size  of  the  difference 
depended  on  the  COA  comparison.  Figure  3-3  shows  that  the 
difference  for  the  two  aided  groups  was  significantly  larger 
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SOURCE  OF  VARIATION 
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Aid  (A) 

37.876 
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37.876 

14.929** 

Experience  (E) 

.428 
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.428 

.169 

Aid  X  Experience  (AE) 

1.364 

1 

1.364 

.538 

Subject/AE 

65.966 

26 

2.537 

Within  Subiects 

AOA 

17.824 

2 

8.912 

9.245** 

Aid  X  AOA 

9.807 

2 

4.904 

5.087* 

Experience  x  AOA 

5.331 

2 

2.666 

2.766 

A  X  E  X  AOA 

.828 

2 

.414 

.429 

Subjects  X  AOA/AE 

50.123 

52 

.964 

*  ■  p  <  .01 
**  ■  p  <  .001 

Table  3-3 

ANOVA  TABLE  FOR  DIFFERENCE  BETWEEN 
POSTERIORS  AND  PRIORS 


than  that  for  the  unaided-experienced  group  only  for  COA  com¬ 
parisons  #2  vs  #1  and  for  #3  vs  #4.  Finally,  the  AOA  main 
effect  occurred  because  the  mean  difference  in  the  logarithms 
for  the  assessed  posterior  and  prior  odds  for  #3  vs  #2  was, 
in  general,  significantly  smaller  than  that  for  #2  vs  #1  or 
for  #3  vs  #4. 

The  results  reported  herein  strongly  correspond 
with  those  obtained  for  the  assessed  posterior  odds.  Taken 
together,  they  indicate  that  experienced  and  inexperienced 
analysts  were  better  able  to  discriminate  between  the  most  and 
least  likely  OPPOR  COAs  when  they  worked  with  the  Bayesian  in¬ 
ference  aid  because  they  wore  better  able  than  unaided  analysts 
to  use  the  implications  of  the  messages  to  move  away  from  their 
initial  position.  The  next  section  presents  results  that  indi¬ 
cate  that  this  increased  movement  and,  in  turn,  improved 
performance,  occurred  because  aided  analysts  were  significantly 
less  conservative  with  than  without  the  aid, 

3 . 2  Hypothesis  II »  Reduced  Conservatism  Wi.th  the  Aid 

The  second  hypothesis  guiding  the  study  was  that  aided 
analysts  would  be  significantly  less  conservative  than  unaided 
analysts.  It  was  hypothesized  that  this  reduced  conservatism 
would  result  in  significantly  better  discrimination  between 
the  most  and  least  likely  OPFOR  COAs  because  aided  analysts 
would  be  revising  their  posterior  odds  to  be  more  consistent 
with  the  assessed  likelihood  ratios  for  the  ten  messages  than 
would  unaided  analysts.  It  has  already  been  shown  that  aided 
analysts  were  more  discriminating  than  unaided  analysts.  In 
addition,  it  has  already  been  shown  that  there  were  no  statis¬ 
tically  significant  differences  in  the  theoretical  posterior 
odds  of  aided  and  unaided  analysts.  Indicating  that  aided  and 
unaided  analysts  alike  drew  the  same  implications  from  the 
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data  and  had  similar  likelihood  ratios  for  the  ten  messages. 

All  that  is  now  required  to  confirm  the  second  hypothesis  is  a 
significantly  smaller  difference  between  the  logarithms  for 
the  theoretical  and  actual  posterior  odds  with  than  without 
the  aid,  thereby  indicating  significantly  less  conservatism 
with  the  aid.  Any  difference  between  the  logarithms  for  the 
final  theoretical  and  assessed  posterior  odds  would  indicate 
some  conservatism  with  respect  to  Bayes*  Theorem;  the  size  of 
the  difference  indicates  the  degree  of  conservatism. 

The  theoretical  posterior  odds  are  those  calculated 
from  the  initially  assessed  likelihood  ratios  using  Bayes* 

Theorem.  The  initially  assessed  likelihood  ratios  were  used 
in  the  aided  condition  because  participants  only  assessed  one 
set  of  likelihood  ratios  in  the  unaided  condition.  It  may  he 
argued  that  it  was  inappropriate  to  use  the  initially  assessed 
likelihood  ratios  in  the  aided  condition,  for  the  aid  is  an 
interactive  one  designed  to  facilitate  iteration  between  the 
likelihood  ratios  and  the  posterior  odds;  this  iteration  was 
expected  to  result  in  revisions  in  the  likelihood  ratios  and, 
in  turn,  lower  final  posterior  odds  than  those  calculated 
using  the  initially  assessed  likelihood  ratios.  Nevertheless, 
the  difference  between  the  logarithms  of  the  final  posterior 
odds  and  those  calculated  using  the  initial  likelihood  ratios 
was  used  in  this  study  to  employ  a  measure  of  conservatism 
comparable  for  both  the  aided  and  unaided  conditions.  If,  under 
the  circumstances  favoring  the  unaided  condition,  the  analyses 
still  showed  a  significantly  smaller  difference  between  the 
logarithms  of  the  theoretical  and  assessed  posterior  probabilities 
with  than  without  the  aid,  the  results  would  clearly  indicate  that 
aided  participants  were  influenced  to  be  significantly  leas 
conservative  (i.e.,  more  Bayesian)  when  revising  their  estimates 
of  OPFOR  intent. 


A  one-between-subiects  and  one-within-sub jects 
design  was  used  to  test  whether  there  were  significant  differ¬ 
ences  in  the  mean  difference  of  the  logarithms  of  the  theoreti¬ 
cal  and  assessed  posterior  odds  for  the  aided-experienced , 
aided-J nexperienced ,  and  unaided-inexperienced  groups?  the 
unaided-experienced  group  did  not  estimate  likelihood  ratios 
and,  therefore,  were  not  included  in  the  analysis.  The  three 
groups  represented  levels  on  the  between-sub jects  factor;  the 
three  COA  comparisons  represented  levels  on  the  within-sub jects 
factor.  The  dependent  measure  was  the  difference  between  the 
logarithms  of  the  theoretical  and  actual  posterior  odds  for 
the  three  COA  comparisons. 

Figure  3-4  presents,  for  each  group,  the  mean 
difference  of  the  logarithms  for  the  theoretical  and  actual 
(final)  posterior  odds  for  each  of  the  three  COA  comparisons. 
Conservatism  is  represented  by  positive  numbers,  for  this 
indicates  that  the  theoretical  posterior  odds  were  larger  than 
the  actual  posterior  odds.  All  three  groups  were  conservative 
to  some  extent  since  all  the  numbers  are  positive.  The  larger 
the  positive  number,  the  greater  the  conservatism. 

Table  3-4  presents  the  ANOVA  results  for  these 
data.  The  only  significant  effect  was  for  groups.  Figure  3-4 
shows  that  the  unaided-inexperienced  group  was  significantly 
more  conservative  than  the  two  aided  groups.  This  finding 
confirms  the  second  hypothesis  guiding  this  study,  which  was 
that  the  performance  of  experienced  and  inexperienced  analysts 
alike  would  be  significantly  less  conservative  when  they  worked 
with  an  interactive  Bayesian  inference  aid.  This  reduced 
conservatism  resulted  in  significantly  better  discrimination 
between  the  most  and  least  likely  OPFOR  COAs. 
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*  ■  p  <  . 05 


Table  3-4 

ANOVA  TABLE  FOR  DIFFERENCE  BETWEEN 
THEORETICAL  AND  ACTUAL  POSTERIORS 


3.472* 


2.584 

.1023 


3,3  Summary 


The  analyses  reported  herein  confirmed  both  hypotheses 
guiding  this  research  study.  Experienced  and  inexperienced 
analysts  were  significantly  less  conservative  in  their  proba¬ 
bility  estimates  when  they  worked  with  an  interactive  Bayesian 

f 

inference  aid;  aided  analysts  had  assessed  posterior  probabili¬ 
ties  that  were  significantly  more  similar  to  their  theoretical 
posterior  probabilities  than  did  unaided  analysts.  This 
reduction  resulted  in  significantly  greater  discrimination 
between  the  OPFOR  courses  of  action  considered  most  and  least 
likely  by  a  majority  of  the  analysts;  aided  analysts  gave  AOAs 
#3  and  #2  significantly  higher  posterior  probabilities  (and 
conversely,  AOAs  #1  and  #4  lower  probabilities)  than  did 
unaided  analysts.  Figure  3-!5  provides  a  pictorial  summary  of 
these  results,  All  statistical  analyses  were,  of  course, 
performed  on  the  logarithms  of  the  odds  estimates  for  the 
three  independent  COA  comparisons  #2  vs  #1,  #3  vs  #2,  and  #3 
vs  #4. 
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SUMMARY  OF  PRIOR.  POSTERIOR  AMD 
THEORETICAL  POSTERIOR  PROBABILITIES  FOR  EACH  GROUP 


4.0  ANALYSES  OF  SUMMARY  REPORTS 
AND  BEHAVIORAL  RESPONSES 


In  practice,  tactical  intelligence  analysts  are  not 
required  to  provide  probability  estimates  of  the  relative 
likelihood  of  different  OPPOR  courses  of  action.  What  they 
are  required  to  do,  either  verbally  or  through  routinely 
submitted  summary  reports,  is  to  provide  the  rank  order  of  the 
OPFOR  COAs  and  to  keep  the  commander  informed  of  significant 
OPFOR  activity.  The  following  sections  present  the  results  of 
three  analyses  designed  to  determine  whether  there  were  signi¬ 
ficant  differences  between  aided  and  unaided  analysts  in  the 
final  rank  order  given  to  OPPOR  COAs,  the  quality  of  their 
summary  report,  or  their  behavioral  responses  to  the  messages. 

4.1  Group  Differences:  Rank  Order  of  OPFOR  Courses  of  Action 

A  two-between-subjects  and  one-within-subjects  ANOVA 
design  was  used  to  test  whether  the  four  groups  assigned 
different  rank  orders  to  the  OPFOR  COAs  when  writing  their 
summary  reports  regarding  OPFOR  activity.  The  two  between- 
subjects  factors  were  the  level  of  experience  and  whether  or 
not  the  aid  was  used  by  the  analyst?  the  one  within-subjects 
factor  was  the  three  COA  comparisons  (#2  vs  #1,  #3  vs  12,  and 
#3  vs  #4).  The  dependent  measure  was  the  difference  between 
the  rank  orders  for  the  COA  comparisons,  which  is  independent, 
as  required  for  an  ANOVA.  The  rank  order  for  the  four  avenues 
of  approach  could  not  be  used  because  it  is  not  independent? 
the  rank  of  the  fourth  avenue  of  approach  can  be  determined  by 
knowing  the  ranks  for  the  other  three. 
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Figure  4-1  presents  the  mean  difference  in  the  rank 
orders  given  the  three  COA  comparisons  by  each  of  the  four 
groups.  When  coding  the  data,  the  rank  orders  assigned  by  the 
analysts  were  Inverted  so  that  more  likely  avenues  of  approach 
were  assigned  higher  ranks.  As  a  result,  positive  numbers  in 
Figure  4-1  indicate  that  the  first  member  of  the  COA  pair  was 
ranked  as  more  likely  than  the  second  member  of  the  pair; 
negative  numbers  indicate  that  the  second  member  of  the  pair 
was  more  likely.  On  the  basis  of  the  results  presented  in 
Section  3.0  for  the  final  assessed  posterior  odds,  one  would 
expect  to  see  positive  numbers  for  all  three  comparisons, 
since  #2  was  more  likely  than  #1,  #3  was  more  likely  than  #2, 
and  #3  was  more  likely  than  #4. 

Table  4-1  presents  the  results  of  the  ANOVA  for  the  rank 
order  differences  assigned  to  the  three  COA  comparisons  by 
each  group,  There  were  significant  main  effects  for  the  Aid 
and  for  the  AOA.  The  Aid  main  effect  occurred  because  the  two 
aided  groups  had  similar,  yet  significantly  different  mean 
rank  differences  from  the  two  unaided  groups.  This  difference 
is  particularly  noteworthy  for  the  #3  versus  #2  comparison. 

The  aided  groups  have  positive  numbers  indicating  that  avenue 
of  approach  43  was  ranked  more  likely  than  #2;  in  contrast, 
both  unaided  groups  have  negative  numbers  indicating  that  they 
ranked  #2  as  more  likely  than  43.  The  AOA  main  effect  occurred 
because  the  mean  rank  order  difference  for  the  #3  versus  4? 
comparison  was  significantly  smaller  than  that  for  the  other 
two  comparisons,  a  finding  that  directly  corresponds  to  the 
results  presented  in  Section  3.0. 

Taken  together  with  previous  findings,  these  results 
indicate  that,  in  general,  aided  analysts  not  only  had  different 
final  posterior  probabilities  for  the  OPFOR  COAs  than  did 
unaided  analysts,  they  had  different  rank  orders  for  them  too. 
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Difference 


Avenues  Of  Approech 


Notation 


o 


Aided'Experlenced 

Aided-lnexperienced 

Uneided-Experienced 

Unelded-lnexperienced 


Piguro  4-1 

RANK  ORDER  DIFFERENCES 


t 


t 


Source  of  Variation 

SS 

df 

MS 

F 

Between  Subjects 


Aid  (A) 

6.160 

1 

6.160 

4.688* 

Experience  (E) 

.145 

1 

.145 

.110 

Aid  X  Experience (AE) 

.860 

1 

.860 

.654 

Subject/AE 

34.176 

26 

1.314 

Vfithin  Subiecta 

AOA 

42.650 

2 

21.325 

13.687** 

Aid  X  AOA 

2.636 

2 

1.318 

.846 

Experience  x  AOA 

1.194 

2 

.597 

.383 

A  X  E  X  AOA 

.028 

2 

.014 

.009 

Subjects  X  AOA/AE 

81.004 

52 

1.558 

*  ■  p  <.05 

**  ■  p  <,001 


Table  4-1 

ANOVA  TABLE  FOR  THE  RANK  ORDERS 
GIVEN  THE  OPPOR  COURSES  OF  ACTION 
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In  particular,  aided  analysts  ranked  AOA  #3  as  most  likely, 
while  unaided  analysts  ranked  AOA  as  most  likely.  It  is 
important  to  remember  that  aided  and  unaided  analysts  alike 
estimated  a  higher  theoretical  posterior  probability  for  AOA 
#3  than  that  for  the  other  three  AOAs  combined.  This  clearly 
indicates  that  both  aided  and  unaided  analysts  thought  that 
the  messages  strongly  supported  AOA  #3.  Yet,  only  aided 
analysts  were  able  to  convey  this  implication,  in  terms  of 
their  final  rank  orders  and  posterior  probabilities. 

4 . 2  Quality  of  Summary  Reports 

Six  evaluators  scored  the  summary  reports  on  a  one-to- 
seven  rating  scale,  where  higher  numbers  indicate  better 
reports.  The  evaluators  were  told  to  consider  the  report's 
clarity  and  completeness  in  addition  to  the  quality  of  its 
argument  supporting  the  final  rank  order  when  making  their 
ratings.  The  evaluators  did  not  know  which  AOAs  were  most  or 
least  likely;  consequently,  the  analyst's  rank  order  could  not 
affect  the  evaluator's  rating. 

A  one-between-sub jects  and  one-within-subjects  AN’OVA 
design  was  used  to  evaluate  statistically  whether  the  ratings 
differed  for  the  two  groups  of  inexperienced  analysts.  The 
two  groups  represented  the  between-sub jects  factor  and  the  six 
raters,  all  of  whom  were  familiar  with  tactical  intelligence 
analysis  and  all  of  whom  independently  evaluated  the  quality 
of  the  summary  reports,  represented  the  within-sub jects  factor. 
The  two  groups  of  experienced  analysts  were  not  included 
because  most  of  them  participated  after  the  group  of  raters 
had  been  assembled  by  the  research  team. 

The  mean  rating  for  the  aided-inexperienced  group  was 
3.645.  The  mean  rating  for  the  unaided-inexperienced  ^roup 


was  3.675.  Clearly,  there  were  no  significant  differences  in 
the  quality  of  their  summary  reports. 

4 . 3  Behavioral  Responses 

Four  issues  were  addressed  for  the  behavioral  questions 
answered  after  each  message.  Three  of  the  four  issues  focused 
on  whether  or  not  there  were  significant  differences  in  the 
proportion  of  participants  within  each  group  who,  within  the 
first  five  messages  (1)  requested  a  special  briefing  with  the 
Division  commander,  (2)  decided  that  the  OPFOR  had  selected  a 
primary  avenue  of  approach,  and  (3)  concluded  that  an  OPFOR 
offensive  was  imminent.  The  statistical  significance  of 
differences  in  the  four  groups'  responses  regarding  each  of 
these  three  issues  was  assessed  by  calculating  confidence 
intervals  around  the  proportions  for  each  group*  two  groups 
would  be  significantly  different  if  the  proportion  for  ors 
group  did  not  fall  within  the  confidence  interval  around  the 
proportion  for  the  other  group. 

The  fourth  issue  was  the  relationship  betwsen  requests 
for  further  information  and  the  magnitude  of  the  likelihood 
ratios.  It  was  found  that  participants  almost  always  asked 
for  more  information.  This  issue  was  dropped  from  the  analysis, 
however,  because  the  researchers  concluded  that  the  "Yes-No" 
choice  did  not  provide  a  scale  with  sufficient  discrimination 
to  correlate  effectively  information  requests  with  likelihood 
ratios. 

Table  4-2  presents  the  results  for  the  three  behavioral 
issues  considered  in  the  analysis.  None  of  the  differences 
were  statistically  significant?  consequently,  it  must  be 
concluded  that  there  were  no  significant  differences  in  the 
groups'  behavioral  responses. 


41 


I 

L.  Proportion  of  analysts  whose  first  request  for  a 
special  briefing  with  the  Division  Commander  was 
within  the  first  five  messages: 

Aided-Experienced  (5  out  of  5)  100% 

Aided-lnexperienced  (6  out  of  8)  75% 

Unaided-Experienced  (3  out  of  4)  75% 

Unaided-Inexperienced  (5  out  of  10)  50% 

2.  Proportion  of  analysts  who  specified  a  primary  AOA 
within  the  first  five  messages: 

Aided-Experienced  (2  out  of  5)  40% 

Aided-lnexperienced  (4  out  of  8)  50% 

Unaided-Experienced  (1  out  of  4)  25% 

Unaided-Inexperienced  (5  out  of  10)  50% 

3.  Proportion  of  analysts  who  conclude  within  the  first 
five  messages  that  an  attac]c  is  imminent  within  24 
hours: 

Aided-Experienced  (0  out  of  5)  0% 

Aided-lnexperienced  (1  out  of  8)  12.5% 

Unaided-Experienced  (0  out  of  4)  0% 

Unaided-Inexperienced  (0  out  of  10)  0% 


Table  4-2 

RESPONSES  TO  BEHAVIORAL  QUESTIONS 
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5.0  INDIVIDUAL  DIFFERENCES  IN 
USING  THE  AID 


The  results  presented  In  Section  3.0  indicated  that,  in 
general,  tactical  intelligence  analysts  were  less  conservative 
in  their  final  assessed  posterior  odds  estimates  when  they 
worked  with  an  interactive  Bayesian  inference  aid.  There 
were,  however,  considerable  differences  in  the  way  that  indivi¬ 
dual  analysts  used  the  aid.  In  particular,  the  fifteen  analysts 
in  the  aided  groups  could  be  classified  into  four  categories: 

(1)  those  who  acted  completely  Bayesian  in  revising  their 
posterior  probabilities,  (2)  those  who  were  slightly  conserva¬ 
tive,  (3)  those  who  were  extremely  conservative,  and  (4)  those 
who  were  in  some  sense  radical,  for  they  made  changes  in  the 
posterior  probabilities  for  some  messages  that  were  so  different 
from  the  theoretical  posterior  probabilities  that  they  often 
resulted  in  a  different  final  rank  order  for  the  OPPOR  COAs. 

Each  category  is  discussed  in  turn  below. 

5.1  Bayesian  Analysts 

One  experienced  analyst  and  two  inexperienced  analysts 
who  used  the  aid  acted  completely  Bayesian.  The  experienced 
analyst  accepted  the  theoretical  posterior  probabilities  for 
each  message;  the  two  inexperienced  analysts  made  minor  changes 
in  the  theoretical  probabilities  for  a  few  messages. 

Figure  5-1  shows  the  theoretical  and  actual  posterior 
probabilities  for  one  of  the  inexperienced  analysts;  all 
inexperienced  analysts  were  coded  using  the  prefix  S.  The 
probabilities  are  represented  on  the  ordinate,  and  the  messages 
are  identified  on  the  abscissa,  A  plus  sign  on  the  abscissa 


indicates  modified  posterior  probabilities,  thereby  permitting 
one  to  see  how  they  were  changed  from  the  initially  calculated 
posterior  probabilities.  Figure  5-1  shows  that  analyst  S16 
made  only  one  change  in  the  posterior  probabilities,  and  that 
was  to  lower  slightly  the  probability  for  AOA  #3  and  correspond¬ 
ingly,  to  raise  slightly  the  probability  for  AOA  #4  after  the 
fifth  message.  This  change  had  practically  no  effect  overall, 
for  the  actual  posterior  probabilities  after  the  tenth  message 
were  essentially  identical  to  the  theoretical  ones.  As  a  result, 
S16  has  been  classified  as  completely  Bayesian. 

5 . 2  Slightly  Conservative  Analysts 

Three  experienced  and  two  inexperienced  analysts  were 
classified  as  slightly  conservative  because  their  final  actual 
posterior  probabilities  were  noticeably  different  from  their 
final  theoretical  posterior  probabilities.  This  occurred 
because  of  noticeable  modifications  in  the  theoretical  posterior 
probabilities  for  some  of  the  messages.  These  modifications 
were  slight  and  infrequent,  however,  compared  to  those  of  analysts 
who  were  classified  as  extremely  conservative.  As  a  result,  the 
actual  posterior  probabilities  of  slightly  conservative  analysts 
were  close  to  their  theoretical  posterior  probabilities. 

Figure  5~2  shows  the  theoretical  and  actual  posterior 
probabilities  for  each  message  for  T2,  one  of  the  experienced 
analysts  who  was  slightly  conservative.  T2*s  actual  posterior 
probabilities  are  noticeably  different  from  the  theoretical 
posterior  probabilities.  In  particular,  the  actual  posterior 
probability  for  AOA  #3  is  lower  than  the  theoretical  one  and, 
correspondingly,  the  actual  posterior  probabilities  for  AOAs 
#1,  #2,  and  #4  are  higher  than  the  theoretical  ones.  This 
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S16’S  ACTUAL  POSTERIOR  PROBABILITIES 


TFFOPFTICAL  ANH  ACTUAL  POFTFPXOP  PpnpABILI'^IFP 
FOR  A  SLIGHTLY  CONSERVATIVE  ANALYST 


occurred  because  T2  lowered  the  initial  theoretical  posterior 
probability  for  AOA  #3  for  measagos  #6,  #7,  and  #10,  However, 
because  these  changes  were  atnall,  the  final  actual  posterior 
probabilities  were  close  to  the  theoretical  ones  and  considerably 
different  from  the  prior  probabilities. 

5 . 3  Extremely  Conservative  Analysts 

Two  experienced  and  two  inexperienced  analysts  were 
classified  as  extremely  conservative.  These  analysts  made 
modifications  that  resulted  in  final  assessed  posterior  proba¬ 
bilities  that  were  markedly  different  from  their  theoretical 
posterior  probabilities  and  markedly  similar  to  their  prior 
probabilities.  As  a  result,  the  intelligence  data  that  the 
analysts  themselves  said  carried  implications  for  OPPOR  intent 
had  minimal  (if  any)  effect  on  their  estimates  of  the  relative 
likelihood  of  the  four  OPPOR  COAs,  The  judgments  of  these 
conservative  analysts  was  almost  completely  dominated  by  the 
information  in  the  scenario  regarding  OPPOR  and  U.S.  force  and 
composition  factors,  and  by  terrain  and  weather  considerations. 

Figure  5-3  shows  the  theoretical  and  actual  posterior 
probabilities  for  T3,  one  of  the  experienced  analysts  who  was 
extremely  conservative.  Notice  that  this  analyst  changed  the 
initial  theoretical  posterior  probabilities  for  half  the 
messages,  and  for  two  of  them  (messages  #1  and  #5)  they  were 
changed  more  than  once.  These  changes  occurred  every  time  the 
analyst  assessed  likelihood  ratios  that  spread  out  the  posterior 
probabilitesr  the  analyst  always  moved  them  close  together 
again.  As  a  result,  the  final  posterior  probabilities  hardly 
differed  from  the  prior  probabilities.  Thus,  the  intelligence 
data  had  practically  no  effect  on  the  analyst's  judgment  as  to 
the  relative  likelihood  of  the  OPPOR  COAs. 
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THEORETICAL  POSTERIOR  PROBABILITIES  T3*S  ACTUAL  POSTERIOR  PROBABILITIES 


5 . 4  Radical  Analysts 


One  experienced  and  two  inexperienced  analysts  were 
classified  as  radical  analysts.  These  analysts  made  changes 
In  their  initial  theoretical  posterior  probabilities  that  were 
so  extreme  that  they  often  resulted  In  final  rank  orders  for 
the  OPFOR  COAs  that  were  different  from  those  for  their  theoret¬ 
ical  posterior  probabilities.  These  extreme  changes  were 
often  made  for  only  one  or  two  messages.  When  they  occurred 
late  In  the  session /  they  apparently  represented  a  complete 
reorientation  in  the  analyst's  thinking »  two  of  the  three 
analysts  even  changed  the  rank  order  of  the  two  OPFOR  COAs 
they  considered  most  likely.  Although  this  reorientation  in 
thinking  cannot  be  modeled  by  Bayes'  Theorem,  it  may  well  have 
been  facilitated  hy  working  with  an  interactive  Bayesian 
inference  aid.  Because  they  were  able  to  see  interactively 
the  implications  of  their  judgments  (both  likelihood  ratios 
and  posterior  probabilities)  for  intelligence  data  over  time, 
these  analysts  may  well  have  been  able  to  completely  re-evaluate 
their  position  regarding  OPFOR  intent. 

Figure  5-4  shows  the  theoretical  and  actual  posterior 
probabilities  for  TIO,  one  of  the  experienced  analysts  who 
acted  radically.  Notice  that  the  theoretical  and  actual  rank 
orders — and,  consequently,  posterior  probabilitiee  for  AOA  #3 
and  #4--are  quite  different.  This  difference  occurred  because 
TIO  changed  the  initial  posterior  probabilities  for  AOAs  #3 
and  #4  for  message  #7  from  .38  and  .59,  respectively,  to  .39 
and  .36,  respectively.  AOA  #3's  reduction  of  .23  probability 
points  primarily  benefited  AOA  #2,  which  moved  from  .03  to 
.20>  AOA  #1  moved  from  ,01  to  only  .05.  This  shift  represented 
a  marked  reorientation  in  the  analyst's  thinking.  Prior  to 
the  shift,  the  analyst  was  saying  that  the  OPFOR 's  main  offensive 
would  be  only  along  the  western  half  of  the  front  (AOA  14  is 


the  western-most  avenue  of  approach) ,  but  after  the  shift,  the 
analyst  was  saying  that  the  offensive  would  be  considerably 
more  toward  the  center  of  the  front  with  a  distinct  possibility 
of  seeing  it  in  the  eastern  half  along  AOA  #2.  A  noninteractive 
Bayesian  aid,  which  was  represented  by  the  theoretical  posterior 
probabilities,  would  not  reflect  this  reorientation  in  the 
analyst's  thinking.  However,  an  interactive  one  does  reflect 
it  and,  as  suggested  earlier,  may  even  facilitate  it  by  providing 
analysts  with  a  means  of  critically  examining  the  implications  of 
their  judgment. 

5 . 5  Summary 

The  results  presented  in  this  section  indicate  that  there 
were  substantial  differences  in  the  way  individual  anlaysts 
used  the  aid.  The  analysts  were  classified  into  four  categories: 
those  who  acted  completely  Bayesian  in  revising  their  posterior 
probabilities,  those  who  were  slightly  conservative,  those  who 
were  extremely  conservative,  and  those  who  were  in  some  sense 
radical,  for  they  made  changes  in  their  initial  theoretical 
posterior  probabilities  for  some  messages  that  were  so  extreme 
that  they  often  resulted  in  final  rank  orders  that  were  different 
(or  were  close  to  being  different)  from  those  based  on  their 
final  theoretical  posterior  probabilities. 
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6.0  CONCLUSIONS  AND  RECOMMENDATIONS 


I 

This  research  study  demonstrated  that  tactical  Intelligence 
analysis  could  be  Improved  by  letting  analysts  work  with  an 

•  interactive  Bayesian  inference  aid.  Experienced  and  inexperi¬ 
enced  analysts  working  with  the  aid  were  better  able  than 
unaided  analysts  to  differentiate  between  the  two  OPPOR  COAs 
considered  most  likely  and  the  two  OPPOR  COAs  considered  least 

•  likely  by  the  majority  of  analysts.  Furthermore,  analysts 
working  with  the  aid  specified  a  final  rank  order  for  the 
OPPOR  COAs  that  was  more  consistent  with  the  implications  of 
the  data  according  to  the  assessed  likelihood  ratios  of  aided 

•  and  unaided  analysts  alike.  This  improved  performance  occurred 
because  aided  analysts  were  significantly  less  conservative  in 
their  posterior  probability  estimates  than  unaided  analysts. 

>  Although,  in  general,  tactical  intelligence  analysts  were 

less  conservative  with  the  aid,  there  were  considerable  differences 
in  the  way  that  individual  analysts  used  it.  The  fifteen 
analysts  in  the  aided  groups  could  be  classified  into  four 

•  categories;  those  who  acted  completely  Bayesian  in  revising 
their  posterior  probabilities,  those  who  were  slightly  conserva¬ 
tive,  those  who  were  extremely  conservative,  and  those  who 

were  in  some  sense  radical,  for  they  made  changes  in  the 

•  initial  theoretical  posterior  probabilities  for  some  messages 
that  were  so  extreme  that  they  resulted  in  final  rank  orders 
for  the  OPPOR  courses  of  action  that  were  different  (or  were 
close  to  being  different)  from  those  based  on  their  final 

•  theoretical  posterior  probabilities.  The  proportion  of  experi¬ 
enced  and  inexperienced  analysts  in  each  category  was  quite 
similar. 

e 
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There  were  no  significant  differences  in  the  quality  of 
the  summary  reports  written  by  aided  or  unaided  analysts,  or 
in  their  responses  to  the  behavioral  questions.  This  finding 
is  not  surprising.  The  tactical  intelligence  analyst's  job 
has  a  number  of  related  requirements.  These  requirements 
include  keeping  track  of  OPPOR  activity,  correlating  OPFOR 
activity  with  OPFOR  intent,  and  keeping  the  friendly  commander 
informed  of  both.  When  performing  the  latter  requirement,  the 
analyst  is  trained  to  emphasize  the  first  requirement,  that  of 
providing  information  about  OPFOR  activity,  and  not  the  second 
requirement,  that  of  making  inferences  about  OPFOR  intent. 

The  summary  report  and  behavioral  questions  tried  to  simulate 
the  first  requirement;  the  final  posterior  probabilities  and 
rank  orders  of  the  OPPOR  courses  of  action  tried  to  simulate 
the  second  requirement.  There  is  no  reason  to  hypothesize 
that  improved  performance  in  inferring  OPPOR  intent  would 
necessarily  result  in  improved  performance  in  providing  infor¬ 
mation  about  OPPOR  activity,  although  such  a  finding  would,  of 
course,  be  an  extremely  important  one.  There  is,  however, 
reason  to  hypothesize  that  it  would  result  in  significant 
improvements  in  the  intelligence  collection  process  and  in  the 
performance  of  friendly  commanders  by  giving  them  more  time 
for  tactical  decision  making  and  implementation.  Investigation 
of  these  hypotheses  were  beyond  the  scope  of  this  study. 

Future  research  efforts  should  focus  on  replicating  the 
results  reported  herein  with  more  complex  tactical  intelligence 
problems.  In  particular,  complexity  should  vary  in  three 
general  ways:  by  varying  characteristics  of  intelligence 
data,  by  linking  together  tactical  intelligence  analysis  with 
subsequent  data  collection  and  tactical  decision  making,  and 
by  more  generally  representing  the  analysts'  working  environment. 
Each  form  of  complexity  is  considered  in  turn. 
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The  intelligence  data  transmitted  in  the  ten  messages 
used  in  this  study  were  relatively  simple  and  straightforward, 
for  they  were  perfectly  reliable  and  unambiguous.  Actual 
tactical  intelligence  data  are  seldom  this  way.  By  making  the 
data  unreliable,  for  example,  one  Is  conceptually  developing  a 
hierarchical  or  multi-level  Bayesian  inference  problem,  since 
there  are  two  steps  in  the  analyst’s  inference  process.  In 
the  first  step,  the  analyst  must  evaluate  the  probability 
that  the  datum  is  reliable,  i.e.,  true.  In  the  second  step, 
the  analyst  then  must  evaluate  the  extent  to  which  the  datum 
supports  the  different  CPFOR  COAs.  Experimental  evidence  on 
hierarchical  (or  multi-level)  Bayesian  inference  aid  is  unfor¬ 
tunately  scant.  Gettys,  Kelly,  Peterson,  Michel,  and  Steiger 
(1973)  have  conducted  two  relevant  studies,  however,  both  of 
which  demonstrated  the  superiority  of  a  hierarchical,  Bayesian 
inference  aid  over  unaided  inference,  for  task  not  involving 
tactical  intelligence  analysis.  In  addition,  Peterson,  Randall, 
Shawcross,  and  Olvila  (1975)  and  Stewart,  O'Connor,  Frisvold, 
Hoblitzell,  Ragland,  and  Randall  (1980)  have  developed  proto¬ 
typical  hierarchical  Bayesian  inference  aids  for  tactical  and 
strategic  intelligence  analysis,  respectively?  but  the  relative 
effectiveness  of  these  aids  has  not  been  evaluated  in  a  controlled 
research  study  like  the  one  reported  herein. 

Future  research  should  also  investigate  the  relationship 
between  tactical  intelligence  analysis  and  both  subsequent  (1) 
data  collection  and  (2)  tactical  decision  making  and  implemen¬ 
tation.  To  date,  no  research  has  systematically  studied  these 
relationships.  It  is  hypothesized  here  that  analysts  will  be 
able  to  improve  their  data  collection  strategies  with  Bayesian 
inference  aids  because  they  will  be  better  able  to  discriminate 
between  the  most  and  least  likely  OPFOR  courses  of  action. 

Such  discrimination  should  help  analysts  select  collection 
sources  that  more  effectively  monitor  OPFOR  activity.  It 


54 


should  also  provide  the  friendly  commander  with  additional 
time  for  tactical  decision  making  and  implementation.  Many  of 
the  experienced  analysts  who  participated  in  the  study  considered 
this  to  be  the  primary  value  of  using  Bayesian  inference  aids 
in  tactical  Intelligence  analysis.  Research  is  needed  to  test 
these  hypotheses. 

Finally,  future  research  should  try  to  represent  more 
effectively  the  tactical  intelligence  analyst's  working  environ¬ 
ment.  In  their  actual  environment,  analysts  work  in  groups,  and 
they  are  often  barraged  with  intelligence  data  over  a  short  time. 
Although  critics  might  argue  that  such  working  conditions  make 
the  utilization  of  Bayesian  inference  aids  infeasible,  just  the 
opposite  is  argued  here.  Once  participating  analysts  became 
familiar  with  the  aid,  they  were  able  to  use  it  to  evaluate  quickly 
the  messages  in  the  study.  Furthermore,  the  aid  could  be  stream¬ 
lined  to  become  a  more  natural  part  of  the  intelligence  analysis 
process,  and  it  can  be  readily  integrated  into  the  computerized 
intelligence  analysis  systems  being  developed  by  the  U.S.  Army. 
Finally,  the  aid  provides  a  means  for  analysts  (and  commanders) 
to  see  if  they  agree  in  the  implications  they  individually  draw 
from  the  data;  this  will  facilitate,  not  hinder,  the  group's  work. 

The  present  study  demonstrated  that,  for  a  representative 
tactical  intelligence  problem,  analysts  were  better  able  to 
discriminate  between  the  more  and  less  likely  OPFOR  courses  of 
action  when  they  used  an  interactive  Bayesian  inference  aid. 

It  represents  an  important  first  step.  Considerable  future 
research  is  required,  however,  to  determine  whether  this 
finding  will  generalize  to  the  analyst's  working  environment. 
Discussions  with  experienced  analysts  indicated  that  if  it 
did,  it  would  have  significant  implications  not  only  for 
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tactical  intelligence  analysis  and  collection,  but  for  tactical 
decision  making  and  implementation  as  well. 
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OPERATION  JAYHAWK:  CONTROL  AND  COORDINATION 
'  OF  DIVISION  OPERATIONS 

Appendix  1  to  Advance  Sheet.  Gfinernl  and  Special  Situations 
1.  GENERAL  SITUATION 


o.  Maps. 

(1)  Map,  series  USACGSC  250— 140,  Western  United  States,  sheet  1  (St  Joseph— 
Topeka),  edition  1976,  1250, CC..  «map  Ay. 

(2)  Map,  series  USACGSC  tOO— 131,  Western  United  States,  sheet  1  (llAwrence— 
Leavenworth),  edition  1976,  1:100,000  (map  B). 

(3)  Sketch  map.  Sketch  of  Corps  Defense  (app  2  to  adv  sheet)  (map  C). 

6.  Background. 

(1)  During  the  summer  of  1977,  Pakland  and  Eurlandia  reached  an  ‘agreement  that 
established  a  mutual  balance  of/^^ces  between  the  two  countries  and  restricted  the  strength  of 
each  country's  combat  forces. 

(2)  Eurlandia,  with  the  help  of  its  allies,  has  begun  a  program  to^iupgrade  its  armed 
forces  and  achieve  complete  equipment  modernization  prior,  to  implementation  of  the  negotiated 
agreement. 

(3)  Pakland  interprets  this  accelerated  modernization  of  Eurlandia  armed  forces  as  a 
threat  to  its  national  security  and  decides  that  it  must  attack  before  Eurlandia  achieves  a 
deHnite  military  advantage.  Pakland  strategy  is  to  gain  maximum  surprise  by  attacking  as 
soon  as  possible. 

(4)  It  is  anticipated  that  Central  Front,  as  part  of  a  larger  Pakland  force,  will  attack 
Eurlandia's  central  region  to  rapidly  rupture  initial  defenses  and  secure  deep  objcctivfrs. 

2.  SPECIAL  SITUATION 


0.  Allied  forces.  All  units  of  Ist  (US)  and  2d  (US)  Corps  are  located  south  and  east  of  the 
area  of  operations  (off  the  map)  where  they  arc  currently  revising  and  updating  defensive  plans 
in  light  of  an  anticipated  attach  by  Pakland  forces.  The  1st  (US)  Corps  consists  of  the  52d  Mech 
Div,  23d  Armd  Div,  312th  Sep  Mech  Bde,  and  the  201st  Armd  Cav  Rcgt,  and  is  assigned  a 
defensive  sector  west  of  the  Missouri  River.  'Che  corn'*  missinri  is  tn  defc-nd  and  dpstrov  onemv 

eHtautiiih  n  covcnii!;  torrp  in  nmu'a  t’.n'cc.s  in  riv'.m^^nnLiiL-  .'.re-'VJ.Yi  HA).  The  corps 

coWna^nidcr  viauaiuesi"alie[en87ve'V"ntmitrc?np"oying^ITe23aArm7Tuv  urn  Ihe'bii’d  Mech  Div 


from  west  to  east.  Divisions  will  control  the  covering  force.  The  3l2th  Sep  Mech  Bde,  minus  a 
tank  and  mechanized  battalion,  will  be  ottached  to  the  23d  Armd  Div  since  this  di\’ision  sits 
astride  the  enemy’s  best  avenue  of  approach  into  the  corps  sector.  The  201  tH  Armd  Cnv  Urgt  (~) 
with  one  squadron  has  been  attached  to  the  52d  Mech  Div  to  assist  that  unit  in  accomplishing 
its  covering  force  mission.  The  remaining  two  squadrons  of  the  20l8t  Armd  Cav  Regthavo  bc-en 
otinched  to  the  23d  Armd  Div  for  the  same  reason.  More  than  likely  the  312th  Sep  Mech  Ude  (— ) 
will  be  U3ed.,D8  the  control  headquarters  for  the  covering  force  of  the  23d  /Sxmd  Div.  On 
completion  of  the  covering  force  missions,  the  52d  Mech  Div  will  release  the  201st  Armd  Cav 
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Regt  headquarters  and  air  cavalry  troop,  and  the  23d  Armd  Div  will  release  one  of  the  two 
attached  squadrons.  Initially  corps  will  have  two  battalions  in  reserve.  On  completion  of  the 
covering  force  mission,  the  20l8t  Armd  Cav  Regt  with  two  battalions  and  one  armored  cavalry 
•quadron  will  be  corps  reserve. 

b,  62d  Mech  Div,  The  52d  Mech  Div  consists  of  six  mechanized  battalions,  five  tank 
battalions,  and  its  organic  cavalry  squadron.  Additional  assets  provided  by  corps  to  the 
division  include:  the  20l8t  Armd  Cav  Regt  (minus  two  squadrons),  a  corps  artillery  group  (five 
battalions);  an  attack  helicopter  company;  a  Hawk  battalion  (SP);  and  a  combat  engineer 
battalion. 

» 

e.  23d  Armd  Div,  The  23d  Armd  Div  consists  of  six  tank  battalions,  five  mechanized 
battalions,  and  the  organic  cavalry  squadron.  Additional  assets  provided  by  the  corps  to  the 
division  include;  the  312th,  Sep  Mech  Bde,  consisting  of  one  mechanized  battalion,  one  tank 
battalion,  and  two  squadrons  from  the  201st  Armd  Cav  Regt;  a  corps  artillery'  group  (six 
battalions);  an  attack  helicopter  company;  a  Hawk  battaljon  (SP);  and  a  combat  enspneer 
battalion. 

d.  Pakland /orces. 

(1)  Packland  has  deployed  major  forces  into  maneuver  areas  located  30  kilometers  north 
of  the  international  boundary. 

(2)  Current  intelligence  indicates  that  the  attock  will  probably  not  take  place  for 
approximately  2  to  3  days. 

(3)  The  Front  attacking  Eurlcndia’s  central  region  wdll  probably  include  three  armies— 
the  7th  Tk  Army,  6th  and  10th  Gd  Tk  Armies— and  a  motorized  rifle  division  in  reserve.  In  the 
attack,  these  forces  will  likely  be  organized  in  two  echelons,  with  maximum  tank  forces 
employed  forward  to  achieve  a  rapid  breakthrough.  Appendix  3  is  the  Pakland  Central  Front 
troop  list  of  maneuver  units. 

(4)  Pakland  is  expected  to  have  air  superiority  initially;  however,  allied  air  forces  would 
be  able  to  achieve  local  air  superiority  for  limited  periods  of  time.  Indications  are  that  nuclear 
weapons  will  not  be  used  in  the  attack. 

(6)  Additional  intelligence  is  included  in  Annex  B,  Intelligence,  to  Ist  Corps  OPLAN  15. 
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Appendix  3  to  Advance  Sheet.  Pakland  Central  Front  Troop  List  of  Maneuver  Units* 

1.  7th  Tk  Army 

ISthTkDiv 
20th  Tk  DIv 
SlitTkDlv 
SethTkDiv 
80th  Mts  Rifle  Div 
7th  SSM  Bde 
7th  AD  Regt 
7th  Arty  Bde 
7th  Engr  Regt 
7th  Pon  Brg  Regt 
7th  Ailt  Crossing  Bn 

2.  8th  Od  Tk  Army 

9th  Od  Tk.Dlv 
10th  Tk  Div 
83d  Gd  Tk  Div 
6th  Mtz  Rifle  Div 
8th  SSM  Bde 
8th  AD  Regt 
8th  Arty  Bde 
8th  Engr  Regt 
8th  Pon  Brg  Regt 
8th  Aslt  Crossing  Bn 

8.  10th  Tk  Army 
24th  Gd  Tk  Div 
26th  Tk  Div 
81st  Mtz  Rifle  Div 
10th  SSM  Bde 
10th  AD  Regt 
10th  Arty  Bde 
10th  Engr  Regt 
10th  Pon  Brg  Regt 
10th  AsU  Crossing  Bn 

4.  83d  Mtz  Rifle  Div 

NOTE:  This  is  a  partial  troop  list. 


t 
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Appendix  6  to  Advance  Sheet.  Extract  of  Corps  Analysis  of  Area  of  Operations 

(Classification) 

STATES  (ST  JOSEPH— TOPEKA)  , 


*  A  * 


A  A  A  A  A  A  A 

b .  Terrain. 

(1)  Relief  and  drainage  systema.  The  area  under  study  consists  of  a 
plateau  that  varies  in  elevation  from  305  to  366  meters.  Two  low  ridges  run 
generally  northwest  to  southeast •  one  m:.away  between  STRANGER  Creek  and  the 
MISSOURI  River  and  one  between  STRANGER  Creek  and  the  DELAWARE  River.  Another 
ridge  runs  ,<^enerally  north  to  south  along  highway  75  between  the  DELAWARE 
River  and  SOLDIER  Creek.  Changes  in  relief  ace  grodual  except  in  etreambeds 
and  along  tha  banks  of  the  MISSOURI  River.  The  principal  etraams  within  the 
area,  other  than  the  rivers,  are  STRANGER  Creek,  SOLDIER  Creek,  VERMILLION 
Creek,  and  CROSS  Creak,  which  flow  south  into  the  KANSAS  River.  The  KANSAS 
River  flows  through  s  wide,  flat  valley  in  a  f lat>bottomad  channel  with  neer* 
ly  vertical  sandy  banka.  The  channel  from  the  junction  of  the  DELAWARE  River 
to  the  junction  with  the  MISSOURI  River  averages  165  meters  wide  and  2  maters 
deeps  ^bs  current  velocity  is  .85  meter  per  second.  The  MISSOURI  River  moan* 
ders  through  a  wide,  flat  volley  with  high  bluffs  along  asch  side  of  the 
valley.  The  river  averages  140  meters  wide  in  our  area,  and  though  the  depth 
varies,  a  2-neter‘<deep  channel  is  kept  open;  Ita  current  velocity  is  1.6 
meters  per  second.  The  DELAWARE  River  flows  in  a  V-ahapod  channel  with  rela¬ 
tively  steep  clay  banks.  Thu  average  width  of  the  river  et  VALLEY  FALLS 
(TP8858)  is  40  meters  and  the  depth  is  0.65  meter.  PERRY  LAl'E  dam  waa  de¬ 
stroyed  several  years  ago  and  the  lake  ia  m  longer  a_^  major  water  obstacle . 

f he  DELawaR^  River  continues  south  thTough  the  now  empty  lake  basin  until  it 
Joins  the  KANSAS  River  west  of  LAWRENCE. 

(2)  Vegetation.  The  vegetation  in  the  area  includes  row  crops,  graz¬ 
ing  land,  orchards,  and  wooded  areas.  The  principal  crops  are  whea*:  and  corn. 
The  fields  are  bare  after  harvesting  and  their  surfaces  broken.  Deciduous 
trees  grow  along  the  river  and  creekbeda  and  on  the  slopes  of  the  ridges. 

The  tops  of  the  rldgelinea  ere  generally  clear  of  heavy  vet^station.  Most  of 
the  roads  in  the  area  are  bordered  by  treelines.  The  vide,  ope  fields  in  the 
area  are  separated  in  many  Instances  by  hedges  of  osage  orange  treea. 

(3)  Surface  materials.  Generally,  the  beds  and  banks  of  the  streams 
are  composed  of  finc-textured  clay-type  soil.  The  ridges  are  composed  of 
medium-textured  limestone- type  soil.  The  cultivated  areas  are  characterized 
by  loose  topsoil.  Along  the  ridges,  the  ground  water  level  it  more  thnn  O.hO 
meter  below  the  surface,  and  the  ground  dries  quickly  even  after  heavy  rains. 

I  -  .  -  I, 

(Claaaif icaclon) 


Reference:  Map,  aeries  250-140,  WESTERN  UNITED 
;■  edition  1976,  1:250,000  (map  A). 
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2.  GENERAL  DESCRIPTION  OF  THE  AREA 
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Drying  of  the  soli  is  aided  by  the  frequent  moderate-to-hlgh  winds  that  also 
deposit  soil  dust  on  the  surface  throughout  the  area. 


(4)  Manmade  features.  The  north-south  roads  In  the  area  Include 
Highway  73  from  ATCHISON  through  LEAVENWORTH  and  then  south  to  Highway  24-40; 
Highway  159  from  vicinity  EFFINGHAM  south  through  NORTONVILLE,  joining  Highway 
59  from  ATCHISON,  southwest  through  NORTONVILLE  and  OSKALOOSA  to  Highway  24 
near  the  KANSAS  River;  Highway  75  leading  south  out  of  SASETHA  through  HOLTON 
to  TOPEKA;  and  Highway  63  from  PAI^Eie  CITY  through  SENECA  to  ST  MARY'S,  where 
It  joins  Highway  24.  There  are  five  primary  east-west  roads  in  the  area:  High¬ 
way  4-116  from  ATCHISON  southwest  to  STRANGER  Creek  and  then  west  across  the 
DELAWARE  River  to  HOLTON,  where  it  Joins  Route  16  and  moves  west  out  of  the 
corps  sector;  Highway  7-192-4-16  from  LEAVENWORTH  through  EASTON  and  VALLEY 
FALLS  to  HOLTON  where  they  join  116.  Highway  92  from  LEAVENWORTH  to  OSKALOOSA 
across  the  DELAWARE  River,  where  it  Joins  Route  4  and  turns  south  to  TOPEKA. 
Highway  24-40  from  KANSAS  CITY,  KANSAS,  through  TONGANOXIE,  LAl^ENCE ,  along 
the  KANSAS  River  to  TOPEKA,  and  west  to  MANHATTAN;  end  the  K^NSAS  Turnpike, 
which  extends  from  KANSAS  CITY,  KANSAS,  to  LAkTlENCE,  TOPEKA,  and  southwest  to 
EMPORIA.  All  these  roads  are  two-way,  hard-surfaced  construction.  In  addition, 
loose-surfaced  secondary  roads  and  some  paved  state  and  county  roads  cross  the 
area  generally  from  north  to  south  and  cast  to  west.  Bridges  on  main  highways 
are  usually  two-way  class  60.  Bridges  on  the  secondary  and  farm  roads  are 
usually  one-way  and  seldom  exceed  class  IS.  One  railroad  runs  north  and  south 
from  ATCHISON  along  the  MISSOURI  River  to  LEAVENWORTH.  It  then  branches  south¬ 
east  along  the  MISSOURI  River  to  KANSAS  CITY,  KANSAS,  couth  to  BONNER  SPRINGS, 
iouthwest  to  LAVmENCE,  and  west  through  TOPEKA  to  MANHATTAN.  From  ATCHISON 
another  system  curves  southwest  to  vicinity  STRANGER  Creek  where  one  branch 
turns  west  to  the  DELAWARE  River  and  northwest  along  the  international  bound¬ 
ary,  and  the  other  continues  southwest  to  VALLEY  FALLS  and  TOPEKA.  The  Chi¬ 
cago,  Rock  Island,  and  Pacific  Railroad  runs  southeast  out  of  PAI^EE  CITY  to 
SABSTHA  and  HORTON  where  it  turns  southwest  to  HCLTCN  leading  directly  to 
TOPEKA  and  further  west-southwest  to  HUTCHINSON.  The  Union  Pacific  Railroad 
runs  southeast  out  of  MARYSVILLE  to  FRi\.NKFORT  and  TOPEKA,  then  east  to  LAW¬ 
RENCE  and  KANSAS  CITY.  The  Atchison,  Topeka,  and  Santa  Fe  Railroad,  depicted 
on  current  maps  as  running  northwest  out  of  LEAVENWORTH,  is  no  longer  in 
existence. 


******* 


3.  MILITARY  ASPECTS  OF  THE  AREA 
a.  Tactical  Aspects . 

(1)  Concealment  and  cover. 

(a)  Relief.  Fair  concealment  and  some  cover  from  direct-fire 
weapons  are  provided  by  the  ridges  and  folds  In  the  ground.  In  general,  creek 
and  atrenm  valleys  provide  excellent  cover  'throughout  the  area.  Though  the 
principal  ridges  run  northwest  to  southeast,  the  Highway  92  rldge-PILOT  KNOB 
complex  affords  defilade  In  the  north-south  direction. 

(b)  Vegetation.  The  tops  of  the  principal  ridges  are  generally 
bare  and  will  provide  only  limited  concealmant.  On  the  other  t)and,  the  slopes 
of  many  ridges  are  heavily  wooded  and  provide  excellent  concealment.  Typical 
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examples  of  these  wooded  areas  are  the  western  slopes  of  RIDHE  Road  Ridge 
(Highway  17)|  the  northern  slopes  of  Highway  92  ridge,  and  slopes  of  the  many 
streams.  The  vegetation  will  tend  to  hold  fog,  smoke,  and  chemicals  In  the 
area.  The  natural  concealment  available  In  the  valleys  and  along  the  ridges 
can  be  enhanced  by  the  use  of  smoke. 

(2)  Observation  and  fire  (to  be  determined  by  the  student). 

*  A  •*  *  A  A  A 

(b)  Relief.  Because  there  are  few  major  terrain  features  in  the 
area,  longorange  observation  is  generally  limited,  The  areas  east  and  west  of 
STRANGER  Creek,  especially  along  the  bluffs  overlooking  the  MISSOURI  River, 
afford  excellent  observation  over  adjacent  lowlands.  In  the  LEAVENWORTH  area, 
Highway  92  ridge  and  PILOT  KNQB  provide  good  observation. 

(c)  Vegetation.  Wooded  areas,  tree-lined  roads,  and  a  hedgellkc 
tree  line  bordering  the  fields  restrict  fields  of  fire  and  observation 
throughout , the  area,  The  heavily  wooded  areas  on  the  western  slopes  of  RIDGE 
Road  Ridge  and  the  eastern  slopes  of  the  high  ground  west  of  STRANGER  Creek 
restrict  observation  Into  the  STRONGER  Creek  valley.  Effective  observation  la 
available  from  points  located  along  the  lower  parts  of  these  slopes.  Observa¬ 
tion  from  Highway  92  ridge  Is  also  restricted  to  a  degree  by  some  areas 
partially  defiladed  by  hedges  end  tree  lines.  The  vegetation  in  the  valleys 
and  ravines  will  tend  to  hold  fog  and  smoke  for  Increased  periods  of  time 
thus  reducing  observation  Into  these  areas. 

(d)  Surface  materials.  The  loose,  dry  soli  on  the  ridges  will 
produce  large  dust  clouds  that  will  persist  for  periods  up  to  1  hour  follow¬ 
ing  movement  of  largo  numbers  of  vehicles, 

(e)  Manmade  features.  Some  of  the  taller  buildings  In  built-up 
areas  provide  excellent  observation  Into  surrounding  areas.  Other  structures, 
such  as  water  towers  and  the  grain  elevators  throughout  the  area,  afford 
excellent  observation. 

(3)  Obstacles, 

(a)  Relief.  The  rivers  and  streams  In  the  area  constitute  the 
only  obstacles.  Of  major  concern  are  the  MISSOURI  and  KANSAS  Rivers  and  of 
lesser  Importance  are  STRANGER  Creek,  SALT  Creek,  DELAWARE  River,  SOLDIER 
Creek,  VERMILLION  Creek,  and  CROSS  Creek,  The  highway  and  railroad  bridges 
over  these  rivers  are  Important  to  corps  operations  in  the  area.  Highway 
bridges  are  class  60  on  must  primary  roads  and  are  class  10  to  IS  on  second¬ 
ary  roads.  Railroad  bridges  will  take  any  division  load.  Highway  and  rallcond 
bridges  span  the  MISSOURI  River  at  ATCHISON  and  ST  JOSEPH.  A  highway  bridge 
and  a  railroad  bridge  span  this  river  at  LEAVENWORTH.  Sliould  these  bridges 
be  destroyed,  a  major  engineer  effort  will  be  needed  to  provide  adequate 
bridges.  The  KANSAS  River  has  two  highway  bridges  at  LAI^ENCE,  one  highway 
bridge  at  EUDORA,  one  highway  bridge  at  DE  SOTO,  two  highway  and  two  railroad 
bridges  at  TOPEKA,  one  railroad  and  one  highway  bridge  ot  DONNER  SPRINGS,  a 
highway  bridge  at  TURNER,  and  eight  highway  and  two  railroad  bridges  in  the 
vicinity  of  KANSAS  CITY,  KANSAS.  STRANGER  Creek  is  cros,scd  by  numerous  high¬ 
way  bridges. 
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(b)  Vegetation.  The  wooded  areas  will  Impede  cross-country 
vehicular  movement. 

(4)  Key  terrain  features. 

(a)  Hill  1250  (Qnn?«52). 

(b)  Hill  1088  (TPSfiO) . 

(e)  Hill  1085  (W37). 

(d)  High  ground  *vxci.nlty  TP7243. 

•(e)  Hill  1149  (UP1441). 

Vf)  Hill  1162  (TP9841). 

(g)  Bridges  over  the  KANSAS  River  from  vicinity  of  BONNER 
SPRINGS  (UP3724)  to  vicinity  of  ST  MARTS  (QU5442). 

(5)  Avenues  of  Approach. 

(a)  Available  to  PAKLAND  forces  to  the  KANSAS  River.* 

1.  Avenue.  1:  ATCHISON  (unBR2)  ,  along  Highway  73,  LEAVEN¬ 
WORTH  (UP3354),  along  Highway  7,  to  BONNER  SPRINGS  (UP3724). 

g.  Avenue  2j  HURON  (TP9890) ,  KORTONVILLE  (TP9965)  ,  OSKA- 
LOOSA  (UP0042)  to  crossing  sites  adjacent  to  LAl^NCE  (UP061S).^ 

g.  Avenue  3t  Highway  75  vicinity  of  the  EURLANDTA  border 
(TP6593),  HOLTON  (TP6572) ,  along  HigTway  75,  to  TOPEKA  (TP6825) . 

4.  Avenue  4t  CORNING  (QU5593) ,  ST  CLERE  (QU5462),  to  St 
MARTS  (QU5442).  “ 

******  * 

4.  EFFECTS  OF  CHARACTElllSTICS  OF  THE  AREA  (to  be  determined  by  the  student) 

a.  Effect  on  Enemy  Courses  of  Action. 

b .  Effect  on  Own  Courses  of  Action. 

5.  GENERAL  WEATHER  SU>0tARY  (J^  September— 15  October) 

September  through  mid-October  is  a  generally  fair  and  mild  period.  Mod¬ 
erate  temperatures  occur  as  the  warm  and  humid  summer  airmass  begins  to  sub¬ 
side.  Extended  poriods  (10 — 14  days)  of  dry  nnd  fair  weother  will  occur,  with 
temperaturea  ranging  from  a  high  of  75  to  80  .  The  mojority  of  rainfall  occur-, 
as  brief  showers  and  thunderstorms;  however,  infrequent  stationary  frontal 
•yateina  may  influence  the  area  toward  the  end  of  the  period,  bringing  pro¬ 
longed  rainfall  (48 — 72  hours).  There  la  less  than  a  5-percent  probability  of 
■now  during  the  period. 


SPECIFIC  WEATHER  PARAMETERS 

a.  Tompurnturc  C°F): 

>lenn  D.itly  Max 
Sep/Oct 

FORT  LEAVENWORTH  80/72 

TOPEK<\  80/70 

fUNllATTAN  81/72 


Me.nn  D.-^lly  Min 
Sep/Oct 


55/45 

58/47 

57/46 


Extreme  Min 
Si-'p/Oct  ” 

28/19 

38/24 

31/20 


Extroiiic  Max 
Scp/Uct 

110/98 

104/96 

112/100 
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b.  Humidity 

(Sep/Oct)  (Percentage) 

e 

e 

' 

0400 

1300 

Sep/Oct 

Sep/Oct 

FORT  LEAVENWORTH 

.  89/84 

53/49 

TOPEKA 

79/76 

50/48 

MANHATTAN 

81/80 

50/47 

Cl  Wind  (Sep/Oct)  (Knots): 

• 

Mean  Speed 

Peak  Gust 

Prevalllns  Direction 

Sep/Oct 

Sep/Oct 

Sep/Oct 

FORT  LEAVENWORTH 

6/6 

46/41 

S/S 

TOPEKA 

8/8 

42/47 

8/S 

MANHATTAN 

8/8 

54/55 

e 

S/S 

d.  Prectpltatton  (In)! 


Mean 

24-hr  Max 

Mean  Snowfall 

btax  Snowfall 

Sep/Oct 

Sep/Oct 

Sep/Oct 

Sep/Oct 

FORT  LEAVENWORTH 

3. 1/2.6 

4. 3/3. 4 

0/.05 

0/.05 

f 

TOPEKA 

.  2. 8/2. 3 

2. 8/4. 8 

.05/. 05 

•.05/. 05 

MANHATTAN 

3. 5/2. 6 

3. 7/3. 5 

0/.05 

0/.05 

c.  Ceiling  and  Visibility  (Sep/Oct)  (Percentage  of  time): 

• 

FORT 

( 

LEAVI\r!HOfIH. 

mm 

MANHATTAN 

Sep/Oct 

Sep/Oct 

Sep/Oct 

Less  than  3,000  ft 

and/or  3  mi 

15/12 

11/12 

n/ii 

Less  than  1,500  ft 

and/or  3  mi 

10/8 

7/8 

7/7 

i 

Less  than  1,000  ft 

and/or  2  mi 

6/5 

5/6 

Leas  than  200  ft 

and/or  ^  mi 

1/1 

.5/1 

1/.5 

7.  SOl.AR  ANU  MOON  DATA 


a. 

Solar  Data, 

Fort  Leavenworth  (Valid  for 

20th  Century 

Z.  1 

B^^NT 

(Local 

BHCT 

Standord  Time! 

IB 

! 

SS 

EECT 

EENT 

1  Sep 

0447 

0520 

0548 

1851 

1918 

1951 

2  Sep 

0448 

0521 

0539 

1849 

1917 

1949 

3  Sup 

0449 

0522 

0550 

1B4B 

1915 

1948 

4  Sep 

0451 

0523 

0550 

1846 

1914 

1946 

5  Sep 

0452 

0524 

0551 

.1845 

1912 

1944 

6  Sep 

0453 

0525 

0552 

1843 

1910 

1943 

7  Sep 

0454 

0526 

0553 

1842 

1909 

1941 

8  Sup 

0455 

0527 

0554 

1840 

1907 

1939 

9  Sup 

0456 

0528 

0555 

1833  •* 

1905 

1937 
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10  Sep 

BMNT 

0457 

BMCT 

0529 

IS. 

0556 

1837 

EECT 

1904 

EENT 

1936 

11  Sep 

0458 

0530 

0557 

1835 

1902 

1934 

12  Sep 

0459 

0531 

0558 

1834 

1900 

1932 

13  Sep 

0500 

0532 

0559  . 

1832 

1859 

1931 

14  Sep 

0501 

0533 

•  0600 

1830 

1857 

1929 

15  Sep 

0502 

0534  * 

0600 

1829 

1856 

1927 

16  Sep 

0503 

0534 

0601 

1827 

1854 

1925 

17  Sep 

0504 

0535 

0602 

1825 

1852 

1924 

18  Sep 

0505 

0536 

0603 

1824 

1851 

1922 

19  Sep 

0506 

0537 

0604 

1822 

1849 

1920 

20  Sep 

0507 

0538 

0605 

1821 

1847 

1919 

21  Sep 

0508 

0539 

0606 

1819 

1846 

1917 

22  Sep 

0509 

0540 

0607 

1817 

1844 

1915 

23  Sep 

0510 

0541 

0608 

1816 

1842 

1914 

24  Sep 

0511 

0542 

0609 

1814 

1841 

1912 

25  Sep 

0512 

0543 

0610 

1812 

1839 

1910 

26  Sep 

0513 

0544 

0611 

1811 

1637 

1909 

27  Sep 

0514 

0545 

0612 

1809 

1836 

1907 

28  Sep 

0515 

0546 

0613 

1808 

1834 

1905 

29  Sep 

0516 

0547 

0613 

1806 

1833 

1904 

30  Sep 

0517 

.  0548 

0614 

1894 

1831 

-  1902 

1  Oct 

0517 

0549 

0615 

1803 

182Q 

1900 

2  Oct 

0518 

0550 

0610 

.1801 

1828 

1859 

3  Oct 

0519 

0551 

0617 

1800 

1826 

1857 

4  Oct 

0520 

0551 

0618 

1758 

1825 

1856 

5  Oct 

0521 

0552 

0619 

1756 

1823 

1854 

6  Oct 

0522 

0553 

0620 

1755 

1822 

1853 

7  Oct 

0523 

0554 

0621 

1753 

1820 

1851 

8  Oct 

0524 

0555 

0622  • 

1752 

1819 

1850 

9  Oct 

0525 

0556 

0623 

1750 

1817 

1848 

10  Oct 

0526 

0557 

0624 

1749 

1816 

1847 

11  Oct 

0527 

0558 

0625 

1747 

1814 

1845 

12  Oct 

0528 

0559 

0626 

1746 

1813 

1844 

13  Oct 

0529 

0600 

0627 

1744 

1811 

1842 

14  Oct 

0530 

0601 

0628 

1743 

1810 

1841 

15  Oct 

0531 

0602 

0629 

1741 

1808 

1839 

b •  Moon  Data  ( 1977)  Fort  Leavenworth  (Local  Standard  Tlmn) : 
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Fraction  of  moon. 

Moon  rise 

Moon  set 

illuminated 

September 

1 

2058 

0941 

.89 

2 

2133 

f039 

.82 

3 

2211 

1135 

.74 

4 

2251 

1229 

.65 

5 

2336 

1320 

•56 

6 

1409 

.46 

7 

0024 

1455 

.37 

B 

0115 

1537 

.28 

9 

0210 

1617 

.20 

10 

0308 

1654 

.12 

11 

0409 

1729 

.07 

12 

0511 

1804 

.02 

13 

0615 

1839 

.00 

14 

0720 

1915 

.00 

15 

0827 

1952 

.03 

16 

0934 

2033 

.08 

17 

1041 

2119 

.16 

18 

1147  ■ 

2209 

.25 

19 

1249 

2305 

.36 

20 

1346 

— — 

.47 

21 

1437 

0005 

.58 

22 

1524 

0108 

.69 

23 

1605 

0213 

.79 

24 

1641 

0318 

.87 

25 

1716 

0422 

-.94 

26 

1750 

0525 

.98 

27 

1823 

0627 

1.00 

28 

1856 

0728 

1.00 

29 

1931 

0826 

.97 

30 

2008 

0923 

.93 

October 

1 

2047 

1019 

.87 

2 

2130 

1112 

.80 

3 

2216 

1202 

.72 

4 

2306 

1248 

.63 

5 

2358 

1331 

.54 

6 

____ 

1412 

.44 

*New  noon: 

Moon's 

face 

not  illuminated 

First  quarter: 

One-half  of 

moon's 

face  illuminated 

Full  moon: 

Moon's 

face 

Cully 

illuminated 

Last  quarter: 

Onc-half  of 

noon's 

face  illuminated 

a 


Phase* 


Last  quarter 


Naw  noon 


First  quarter 


Full  noon 


Last  quarter 
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Erflct-lan  ai  csaa 


ober 

tlnued 

Moon  rise 

Moon  set 

UldilliELi 

7 

OOSA 

1AA9. 

.  *  .35 

8 

0153 

1525 

.26 

9 

025A 

1600 

.17 

10 

0356 

1635 

.10 

11 

0501 

1710 

.OA 

12 

0609 

17A8 

.01 

13 

0717 

1828 

.00 

lA 

0827 

1913 

.02 

IS 

0935 

2003 

.06 

ftiftp?* 


New  moon 


8.  WEATHER  FACTORS  OF  OPERATIONAL  SIGNIFICANCE 

••  Severe  Storms;  The  transition  from  the  warm,  moist  summ*»rtime  airmans 
to  Che  cooil  dry,  fall  and  winter  alrmasses  has  the  potential  for  occasional 
■fvere  thunderstorms.  Normally  the  occurrence  of  these  storms  Is  toward  the 
end  of  the  period  when  the  contrast  between  airmasses  becomes  more  acute. 
Operational  planning  should  consider  the  Influence  of  severe  weather.  Timing 
of  movements,  distribution  of  resources,  and  protection  of  personnel  end 
equipment  are  Important  considerations. 

b.  Trafflcabllity.  The  gradual  decrease  in  rainfall  during  the  lato 
summer  an'd  early  fail  results  in  generally  good  trafflcabllity  throughout  the 
region.  Brief  local  restrictions  to  trafflcabllity  may  occur  In  areas  influ¬ 
enced  by  heavy  showers  or  thunderstorms. 

c.  Flying  Weather;  Flying  weather  Is  good  to  excellent  during  the  peri¬ 
od.  OcrasLonal  low  ceilings  and  visibilities  associated  with  frontal  systems 
and/or  local  thunderstorms  may  only  briefly  limit  flying  activities. 
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MESSAGE;  1 


GENERAL  ENEMY  SITUATION 


During  the  past  three  weeks,  PAKLANO  has  deployed  major 
maneuver  and  fire  support  forces  into  traditional  training 
areas  located  approximately  30  kilometers  north  of  the  border. 
The  announced  reason  for  this  move  is  to  conduct  annual 
training  exercises.  PAKLAND  news  media  has  referred  to  this 
as  exercise  HOT  WATER. 

PAKLAND 'S  forces  currently  conducting  this  exercise  have 
been  identified  as  elements  of  the  7th  and  10th  Tk  Armies  and 
the  8th  Gd  Tk  Army,  west  and  north  of  HORTON,  respectively. 

Even  though  these  three  armies  have  moved  to  established 
training  areas,  the  level  of  training  activity  has  been  lower 
than  normally  expected.  PAKLAND  has  been  moving  supplies 
forward  at  an  abnormally  high  rate  during  the  last  three  days 
and  has  been  stockpiling  these  supplies  for  easy  access  of 
all  PAKLAND  forces.  Also  of  significance  is  the  emergence  of 
a  new  control  group  headquarters  located  in  the  area  of  the 
8th  Gd  Tk  Army. 
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ENEMY  ACTIVITIES;  GROUND 

(1)  Activity  level  has  been  less  than  normally  experienced 
during  enemy  training  exercises >  with  the  exception  of  the 

8th  Gd  Tk  Army.  The  2l8t  Tk  Div  HQ  is  currently  communicating 
from  the  vicinity  of  TROY,  which  is  approximately  100  kilometers 
east  of  its  last  known  location  in  the  7th  Tk  Army  area. 

This  information  is  derived  from  COMINT  sources  only. 

(2)  Heavy  road  and  rail  traffic  has  been  observed 
going  into  the  8th  Gd  Tk  Army  area. 

(3)  SIGINT  indicates  that  two  brigades  of  army  group 
artillery  have  moved  into  the  8th  GD  Tk  Army  area.  Air 
reconnaissance  has  not  been  able  to  locate  their  positions. 
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MESSAGE :  3 


ENEMY  ACTIVITIES;  AIR 

Enemy  air  activity  remains  normal. 


MESSAGE:  4 


ENEMY  ACTIVITIES:  OTHER 

Press  releases  have  reported  that  General  Damin,  Com¬ 
mander  >  Pakland  Central  Fronts  will  visit  the  8th  Gd  Tk  Army 
on  September  28th  to  observe  their  training  exercises.  Com¬ 
munications  between  a  mobile  facility  in  the  8th  Gd  Tk  Army 
area  and  suspected  group  headquarters  tend  to  confirm  this 
report. 


MESSAGE:  5 


MISCELLANEOUS 

a.  The  7th  Tk  Army  is  commanded  by  General 
Konef.  He  is  known  to  be  a  studious,  cautious  individual, 
who  believes  in  detailed  planning  and  fights  his  battles  "by 
the  book . " 


b.  The  commander  of  the  8th  Gd  Tk  Army  is 
General  Malenofski,  an  aggressive  commander,  willing  to  take 
chances.  He  very  seldom  fights  in  accordance  with  accepted 
enemy  doctrine. 
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MESSAGE:  ^ 


GENERAL  ENEMY  SITUATION 


Enemy  reccnnalseance  forces  have  moved  out  of  the 
training  areas  and  are  now  actively  conducting  reconnaissance 
operations  from  HORTON  west  to  CENTRALIA.  Communication 
level:  Until  approximately  0200  hours,  there  was  an  increase 
in  reconnaissance-type  traffic  along  the  entire  southern 
PAKLAND  border,  with  most  of  the  concentration  in  the  eastern 
quarter  of  the  corps  rone.  Level  of  activity  within  the 
division  training  areas  has  dropped  below  that  normally 
encountered  during  training  exercises. 


MESSAGE:  7 


ENEMY  ACTIVITIES:  GROUND 

Activity  level  continues  to  be  less  than  normally 
experienced  during  enemy  training  exercises,  with  the  ex¬ 
ception  of  the  8th  Gd  Tk  Army.  Commencing  16  September 
there  appeared  to  have  been  a  buildup  of  forces  in  the 
eastern  zone  of  the  8th  Gd  Tk  Army.  Headquarters,'  10th  Tk 
Div,  and  23d  Gd  Tk  Dlv  have  relocated  eastward  and  are 
presently  held  to  be  10  kilometers  north  of  FALLS  CITY  and 
HIAWATHA,  respectively. 
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MESSAGE: 


8 


ENEMY  ACTIVITIES:  GROUND 

Heavy  road  and  rail  traffic  contlnuaa  to  ba  obsarvad 
going  into  tha  8th  Gd  Tk  Army  araa. 


MESSAGE:  9 

ENEMY  ACTIVITIES:  GROUND 

It  appears  as  if  front,  sacond-achalon  artillery  vuiits 
have  moved  forward, 


MESSAGE:  10 


« 


ELECTRONIC  V7ARFARE 

(1)  Strict  SIGSEC  measures  have  been  imposed  on  the 
7th  Tk  Army  artillery  units. 

(2)  At  190200,  all  enemy  units  went  under  radio 
silence. 


t 
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APPENDIX  B 

COMPUTER  PRINTOUTS  SHOWING  HOW 
ONE  OF  THE  EXPERIENCED  ANALYSTS  USED 
THE  INTERACTIVE  BAYESIAN  INFERENCE  AID 


INTRODUCTION 


THIS  PROGRAM  WILL  BE  PARTICULARLY  HELPFUL  TO  YOU  WHt^N: 

1.  YOU  ARE  USING  IPB  PROCEDURES;  YOU  ARE  CREATING  YOUR  EVENT 
matrix;  YOU  HAVE  COMPLETED  YOUR  TERRAIN,  GB,  AND  WE."  A  THE  R 
ANALYSES. 

2.  YOU  HAVE  SOME  INTELLIGENCE:  THAT  SUPPORTS  MORE!  THAN  ONE  COURSE' 
OF  ACTION. 

THE  PURPOSE  OF  THIS  PROGRAM  IS  TO  HELP  YOU  ANALYZE  THE  REL¬ 
ATIVE  LIKELIHOODS  OF  ALTERNATIVE  ENEMY  COURSES  OF  ACTION  (COAB) . 

PRESS  EXECUTE  TO  CONTINUE, 

PRESS  EXECUTE  TO  CONTINUE. 

THE  PROGRAM  WORKS  BY  BREAKING  YOUR  ANALYSIS  DOWN  INTO  THREE 
PHASES: 

<1>  DEFINING  THE  ALTERNATIVE  C0A8 .  HOW  MANY  ARE  THERE?  WHAT 
ARE  THEIR  NAMES? 

<2)  ASSESSING  THE  LIKELIHOOD  OF  EACH  COA  TAKING  INTO  ACCOUNT 
WHATEVER  INTELLIGENCE  INFORMATION  YOU  HAVE  ON  HAND,  PRIOR 
TO  RECEIVING  ANY  NEW  INTELLIGENCE  INFORMATION,  THIS  PRIOR 
INFORMATION  DEALS  WITH  TERRAIN,  OB,  WEATHER,  AND  SO  FORTH. 

CJ)  ASSESSING  HOW  EACH  PIECE  OF  NEW  INTELLIGENCE  YOU  RECEIVE  IM¬ 
PACTS  THE  ORIGINAL  LIKELIHOODS  YOU  ESTIMATED  IN  STEP  2, 

PRESS  EXECUTE  TO  CONTINUE, 
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MAIN  OPTIONS 


1>  PERFORM  AN  ANALYSIS 

2)  OUTPUT  ANALYSIS  RESULTS 

3)  RETRIEVE  OR  STORE  ANALYSIS  RESULTS 

TYPE  THi;:;  numse;r  of  the  desire;i:i  option:  i 


ANALYSIS  options 

n  i;iEFiNE:  the;  alternative  coas 

2)  ASSESS  INITIAL  COA  LIKELIHOOIiS 

3)  ASSESS  IMPACT  OF  NEiU  INTELLIGENCE  ON  COA  LIKELIHOODS 
TYPE  THE  NIIMDER  OF  THE  DESIRED  OPTION!  1 
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pirii'iys  ihiE  ^!r!iyt!$nyE  ccag 

MOUl  MANY  CDUKSES  DF  ACTION  <COAS)  DO  YOU  IJANT  TO  CONOIOCR? 

HGU  MANY  COURSES  OF  ACTION  <COAS)  DO  YOU  WANT  TO  CONSIDER?  H 

FOR  EACH  COURSE  OF  ACTION  (COA),  TYRE  UP  TO  THIRTY  LETTERS  FOR 
A  LONG  NAME,  AND  UP  TO  THREE  LETTERS  FOR  A  SHORT  NAME. 

COA  1.  (LONG):  “52  MECH  R  SECTOR 

COA  1  (SHORT):  1 

COA  2  (LONG):  :5i2  MECH  C  SECTOR 

COA  2  (SHORT);  2 

COA  Z  (LONG):  2'I!  AR  R  StT.CTOR 

COA  2  (SHORT):  3 

COA  i|  (LONG);  23  AR  L  SECTOR 

COA  L  (SHORT);  L 


ANALYSIS  OPTIONS 

1)  DEFINE  THE  ALTERNATIVE  CO AS 

2)  ASSESS  INITIAL  COA  LIKELIHOODS 

3)  ASSESS  IMPACT  OF  NEW  INTELLIGENCE  ON  CDA  LIKELIHOODS 
TYPE  THE  NUMBER  OF  THE  DESIRED  OPTION:  2 


f'HASt;:  II;  A;iSi':sG  initial  coa  lii'Llihoods 

THE  P(:)Bi5IHLl£  COAB  ARE: 

1)  52  MECH  R  iiECTOR 

2)  52  MECII  C  SECTOR 

3)  23  AR  R  BECiDR 

4)  23  AR  L  BECTGR 

GIVEN  ALL  PRIOR  INTELLIGENCE  UATA  (E . B . ,  TERRAIN  ANALYSIS,  OB, 
UEATHER),  RANK  ORDER  THE  COAS  FROM  MOST  TO  LEAST  LIKELY, 

TYPE  EACH  NUMBER  FDLLDUED  BY  A  SPACE:  2  3  I  i-l- 
HOU  MANY  TIMES  AS  LIKELY  IS  2  AS  COMPARED  IJITH  3  1 

HOIJ  MANY  TIMES  AS  LIKELI  IS  2  AS  COMPARED  UITH  I  ?  2 

HOU  MANY  TIMES  AS  LIKELY  IS  2  AS  COliPARED  UITH  M  ?  3 


ASSESSED  PRIOR  ODDS  I 

2  2  2 
VS  VS  VS 

3  1  M 

1.00  2 ,00  3.00 

BASED  ON  YOUR  ASSESSMEN  TS  OF  I  HE  PRIOR  ODDS,  THE  CALCULATED 
PRIOR  PRODADILITIES  OF  THE  COAS  ARE: 

1.  2  3  'i 

,10  ,35  .35  ,12 

GIVEN  ALL  PRIOR  INFORMATION,  DO  THESE  PRIOR  V^RODADILITIFS 
ACCURATELY  REFLECT  YOUR  CURRENT  PERCEPTION  OF  THE  LIKELIHOODS 
or  YOUR  COAS?  Y 
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f  Qi.i  ii,',yi:.  jiou  cDf-ii'Li;;.  i-iir'iSt;.  ;i:r,  AssEsshLf  T  or  ]  n:i.  i:iai.  Lur, 
iK.iuii':; .  you  siiquUi  nqu  uvaluate;  thi-,  impai'T  of  wiiatfvef  ni:u 
■INTCILLIGLMCi;;  INFQRhATlON  YOU  lilLCILIVt;  ,  THIO  IMI- ORfiAT:!  ON  Ull.L 

take;;  the  form  of  reports  of  obsei'^ved  enemy  ACT;iv:tTiE;;s .  to  make 
Tui;;:!:iE::  AOGFOOMEiihno,  oe;:i...e:(;:t  option  3  from  the  options  WHir:ii  uii.,.i.., 

BE  III  SPLAYED  NEXT, 


PRESS  EXECUTE  TO  CONTINUE, 


ANALYSIS  OPTIONS 

1)  DEFINE  THE  ALTERNATIVE  COAS 

2)  ASSESS  IN:rriAL  CO  A  LIKELIHOODS 

3)  ASSESS  IMPACT  OF  NEU  INTELLIGENCE  ON  CO A  LIKELIHOODS 
TYPE  THE  NUMDER  OF  THE  DESIRED  OPTION:  3 


PHASE  III; 


ASSESS I NO  THE  IMPACT  OF  NEU  INTELLIGENCE 
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PRESS  EXECUTE  TO  CONTINUE. 


0  RI:P0RT<S>  DF  new  intelligence  have  deem  entered  GINCE  YCUR 
AGGEEBMENT  OF  THE  PRIOR  PRDDABILITIY  OF  EACH  CGA , 

PR  eke;  execute  to  continue. 

THE  PRIOR  rnODADILITIEG  OF  THE  COAG  ARE: 

1,  iiJ  3  M 

,10  .25  ,10 

THE  CORRESPONDING  PRIOR  ODDS  ARE: 

n 

.w  4'..  4>%. 

vs  vs  vs 

3  ;l,  1+ 

1.  .  0  0  2 . 0  0  3 , 0  0 

TYPE  A  LONG  <30  LETTER)  AND  A  SHORT  (3  LETTER)  NAME.  FOR  THE, 
OBSERVED  ENEMY  ACTIVITY, 

LONG  name:  general  en  sit 

SHORT  NAME:  Hi 


IN  THE  NEXT  STEP  YOU  WILL  DE  ESTIMATING  THE  LIKELIHOOD  THAT 
YOU  UOULD  OBSERVE  HI  IF  IN  FACT  THE  ENEMY  WERE  ACTUALLY 
PURSUING  A  GIVEN  COA, 

PRESS  EXECUTE  TO  CONTINUE. 
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I'lli^g  MUlii  hIMIllliMmtmml. 


Tur:  POBSIHLE  COAG  are: 


1>  52  ME  Cl  I  R  B  EC  TOR 
I")  5P  MCCII  C  BECTOR 
3)  23  AR  R  BECTOR 
4  )  22  AR  L  BBXrrOR 

GIVEN  ALL  KNOULEOGL  DL  THE  SITUATION  AT  HAND, 

UNDER  UIHICH  CO  A  UOULD  YOU  BE 
MOST  LIKELY  TO  SEE  H:l.  :  2 
SECOND  MOST  LIKELY:  3 
THIRD  MOST  LIKELY:  1 

EOURTH  MOST  LIKELY:  M 

GIVEN  A  LI..,  or  YOUR  KNOULEDGE  Of  THE  SITUATION,  HOU  MANY  TIMES 
AS  LIKELY  IS  HI  IE  THE  ENEMY  HAD  CHOSEN  TO  F'URSUE 

2  RATHER  THAN  3  ?  1 

2  RATTIER  THAN  J.  ?  2 

2  RATHER  THAN  'I  ?  3 

GIVEN  THE  LIKELIHOOD  RATIOS  THAT  YOU  HAVE  PROVIDED,  TTIE 
POSTERIOR  PROBADILITIES  OP  THE  COAG  ARE: 

i.  2  3  E 

,11  ,i|2  ,H2  ,()S 

AND  THE  CORRESPONDING  POSTERIOR  ODDS  ARE: 


' I  'I  '1 

A..  «.  «•.. 


VS 

VS 

VS 

s 

1 

M 

:l  .  0  0 

4  ,  0  0 

9 . 0  0 

DC)  THESE  POSTERIOR  PRODAHILITIEFS  AND  ODDS  ACCURATELY  REFLECT 
YOUR  PEELINGS  CONCERNING  THE  LIKELIMOODS  OP  THE  GOAS?  N 


Tlir  PKIOr-!  PROHAHILITIEIG  FOR  THEE  COAS  BEFOREE  THE  REPORT  OF  Rl 

:l  1?.  2  4- 

.IB  .25  .25  .12 


IflELUrUMG  INI  DRMATION  CONCERNIMG  THEE  CURREMT  DATUM,  THEE  E'ROD- 
Ari];i..:E  Tn::!:;  A!  :i:;  : 

:i.  2  3  >1 

.11  ,M-2  .I.I-2  .05 

PLEEABEE  BPECIFY  A  BEET  OF  POEDTEERIOR  PROBAmi.lTlEEB  TIIAT  ACCURATEE-Y 
REEFL.ECT'E  HDU  LIKEELY  YOU  (EONBIDEE-!  EEACH  COA  TU  BE,  OIVEEN  ALI,.. 

avaie.able:  iNFor'-MATiori , 

TYPEE  YDUE)  DIREECT  ASBEBiaMCENTS  OF  THE  POBTERIGR 
PROBABIEITIEEB  FOR  ALJ...  COAS;  .10  .  3S  .35  .12 


the::  LlKELIHOOr.i  RATIOB  in  thee  FOE.LDUING  table  AREE  GALnULATEri  FROM 
YOUR  i:i]:RlECnT..Y  ABBEEBBEED  POSTERIOR  PROBABILITIEEB ,  THEY  BHOU  HOW 
MANY  TIMEEB  AB  LIKELY  THE  OBSERVED  ENEEMY  ACTIVITY  ( «!  )  UGULB 
BEE,  GIVEN  THAT  THEE  EEtEEMY  HAB  CHOSEN  TO  PURSUE  3  RATHER 
THAN  EACH  OF  THEE  OTHER  COAB . 


PRESS  EXECUTE  TO  CONTINUE 


CALCULATED  LIKELIHOOD  R,'-(TIOE: 


2 

2 

n 

iif 

VS 

"vs 

vs 

3 

1 

>4 

1.00 

.97 

.97 

ASSESS5E1 

[1  LIKELIHOOD  RATIOS! 

'7 

r> 

4m. 

"vs 

"vs 

VS 

3 

1 

4 

1  .  00 

2.0  0 

3,00 

PRESS  EXECUTE  TO  CONTINUE. 


THE  LIKELIHOOD  RATIOS  CALCULATED  FROH  YDIJR  DIRECTLY  ABSEBCED 
POSTERIOR  PRODADILITIEB  ARE  DIFTERE-NT  THAN  YOUR  DIRECTLY 
ASSESSED  LIKELIHOOD  RATIOS  FOR  «1  .  YOU  MAY  WISH  TO  RFVIBE 
YOUR  niREClLY  ASSESSED  POSTERIOR  PRODADILITIES  OR  YOUR  D I  RECTI.. Y 
ASSESSED  LIKELIHOOD  RATIOS  OR  LEAVE  THEM  AS  THEY  ARE.  IT  IB 
NOT  NECESSARY  THAT  YOU  REVISE  EITHER  IF  THEY  DOTH  ACCURATELY 
REFLECT  YOUR  TRUE  FEELINOS. 

PRESS  EXECUTE  TO  CONTINUE. 
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1)  revige:  poste:rior  probahilittes  e"or  the  coab 

2)  RE;;V:r,GE  LIKELIHOOIi  ratiog  for  the  QIiGERVED  E'NEMY  activity  hi 
;j)  REVIGi:  NEITHER 

TYPE.  THE  NUMBER  OF  THE  DEGIREB  OPTION:  2 


SUMMARY  TABlj;: 

FOR 

LXi<i::‘i. 

..IHOOD 

'!> 

As 

*«• 

'1 

VS 

vs 

4 

*"VG 

|i 

ORIGINAL  i', 

00 

1 

2,00 

3.0  0 

CALCULTD  I  . 

0  0 

,97 

.97 

RATIOS: 


YOUR  CURRENT  RANK  FOR  THE  LIKELIHOOD  OF  HI  GIVEN  EACH  COA  IS:  2  '.'S  1 


HAS  YOUR  RANK  CHANGED?  N 


SUMMARY  TAriLE  FOR  LIKFLIIIOOl'i  RATIOS: 


CALCUt..TI' 


VS 

VS 

VS 

3 

1 

>1 

::j  .  0  0 

2 . 0 1) 

1  , 

0  0 

;l  ,  0  0 

.97 

1  . 

0  0 

GIVEN  ALL  Tlir;:  HATA  YOU  HAVE  HAIi  UP  TO  THIS  POINT, 
HOU  MANY  TIMES  AS  LIKELY  IS  THE  CURRENT  DATUM  IF 
THE  F.:Nr.MY  HAD  CHOSEN  TO  PURSUE 


2  rathe:: n  than  :j  ?  i 

2  rather  than  :i  ?  1 

2  RATHER  THAN  'I  ?  1. 


n 

3 

4 

10 

.:j5 

,35 

.12 

10 

.35 

.35 

.12 

10 

.35 

,35 

.  12 

SUMMARY  TADIJ:,  FOR  F'RIOR  AND  POSTERIOR  PRODAi:i:[LITIi:::S : 


PRIOR 


YOUR  ASSESSED  POJ3TERIOR  PRODABILITIEFS  ARE  NEARLY  EQUAL  TO  THE 
CALCULATED  POSTERIOR  PROBABILITIES . 

BO  YOU  UISH  TO  CONTINUE  THE  ANALYSIS  OF  Ml  ?  N 

DO  YOU  UISH  TO  CONTINUE  THE  ANALYSIS  BASED  ON 
ANOTHER.  NEWLY  OBSERVED  ENEMY  ACTIVITY?  Y 
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1  RtlPDIMCB)  OF  NFW  INTFLL JOFNCE  HAVE  rCEN  ENT[{l?Fi:i  SINCIi:  YOUR 
ASOEBOMFHT  OF  THi;;:  PRIOR  PUODADILITIY  OF  EACH  OOA, 

PRESB  FXFCUTE  TO  CONTINUE. 

Tin:  PRIOR  PROIIADILITIES  OF  THE  COAS  ARE: 

1.  3  Ii- 

.10  .30  ,35  .113 

THE  OORRESPONinNO  PRIOR  DMiO  ARE: 


1*  1? 

V3  *"vo  VS 

3  1  1+ 

1,0  0  :l,  ,9L|  2.92 


TYPE.  A  EONS  (30  LETTER)  AND  A  SHORT  (3  LETTER)  NAME  FOR  THE 
ODSERVED  ENEMY  ACTIVITY, 

LONG  NAME!  GROUND  ACTIVITY 
SHORT  name:  «2 


IN  THE  NEXT  STEP  YOU  WILL  BE  ESTIMATING  THE  LIKELIHOOD  THAT 
YOU  WOULD  OBSERVE  «2  IF  IN  FACT  THE  ENEMY  WERE  ACTUALLY 
PURSUING  A  GIVEN  CO  A. 


PRESS  EXECUTE  TO  CONTINUE, 


TUI;;;  bqab 

1.)  ^i2  Mi;;;cii  R  sector 

2)  52  MECM  C  SECTOR 

3)  23  (*tH  R  SECTOR 
1+)  23  Mi  1.  SECTOR 

CilVEN  ALL..  KNOWLEHOE  OF  THE  BITOATION  AT  HAHM, 

UNDER  Uli;i;CH  CO  A  UOULD  YOU  DE 

MOST  to  bee;:  «2  :  2 

BECOMD  MOST  LIKELY;  3 
THIRD  MOST  LIKELY;  1 

rODRTH  HOST  LIKELY:  M- 

0;i;VEN  ALL  OF  YOUR  KNOULEDOE  OF  THE  BITUATHIH,  HDU  HANY  TIMES 
AS  LIKELY  IS  «2  IF  THE  ENEMY  HAD  CHOSEN  TO  PURSUE 

2  RATI-IER  THAN  3  ?  1 

2  RATI'lER  THAN  1  ?  I 

2  RATHER  THAN  'I  ?  ;l 

GIVEN  THE  LIKELIHOOD  RATIOS  THAT  YOU  HAVE  PROVIDED,  THE 
POSTERIOR  PRODADILITIES  OF  THE  COAS  ARE: 

;l.  2  3  i| 

,1S  ,35  .3Lri  ,1.2 

AND  THE  CORRESPONDING  POSTERIOR  ODDS  ARE; 


2  2 

"vs  *'vs  "vs 

3  .1.  4 

1,01)  1,?'+  2.92 


DO  THESE  POSTERIOR  PROBABILITIES  AND  ODDS  ACCURATELY  REELECT 
YOUR  FEELINGS  CDNCERNINO  THE  LIKELIHOODS  OF  THE  COAS?  Y 
DO  YOU  UISH  TO  CONTINUE  THE  ANALYSIS  BASED  ON 
ANOTHER,  NEWLY  OBSERVED  ENEMY  ACTIVITY?  Y 
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APPENDIX  C 

DATA  FORMS  USED  TO  ESTIMATE 
THE  PRIOR  ODDS,  LIKELIHOOD  RATIOS, 

AND  POSTERIOR  ODDS  IN  THE  UNAIDED  CONDITION 


Participant  Number 


Date 


ANSWER  SHEET  FOR  PRIOR  ODDS 


1.  Rank  order  the  AOAa  from  moet  to  least  likely,  based 
upon  the  set  of  background  intelligence  data. 

Most  Likely  AOAt  - - - — 

Second  Moat  Likely  _ _ _ _ _ 

Third  Most  Likely  _ _ 

Least  Likely  AOA  _ _ — — 

2.  how  m.hy  tloo.  a.  UX.ly  th.  moat  Uhely  AOh 

is  versus  each  of  the  other  AOAsi 

(ft)  Most  versus  second  most  likely  AOA 

(b)  Most  versus  third  most  likely  AOA  - — 

(c)  Most  versus  least  likely  AOA  - - — 


1,^ 


C-2 


laiiiiietiaHiiiwii 


i 


Participant  Number  _  Date 

Maaaage  _ 


ANSV7ER  SHEET  FOR  LIKELIHOOD  RATIOS 


1.  Rank  order  the  AOAs  in  terms  of  the  message. 

(a)  AOA  for  which  you  would  be  most  likely  to  receive 
messaget  _______________________________________ 

(b)  Second  most  likely  AOA: _ 

(c)  Third  most  likely  AOA: _ 

(d)  Least  likely  AOA:  _ 

2.  Indicate  how  many  times  as  likely  you  would  be  to 
receive  the  message^  given  the  most  likely  AOA  versus 
each  of  the  other  AOAs. 

Note:  These  are  the  likelihood  ratios  (LR). 

(a)  Most  likely  versus  second  most  likely  AOA:  _ 

(b)  Most  likely  versus  third  most  likely  AOA:  _ 

(c)  Most  likely  versus  least  likely  AOA  _ 
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Participant  Number  _  Date 

Massage  _ 


ANSWER  SHEET  FOR  POSTERIOR  ODDS 


1.  Rank  order  the  AOAs  from  most  to  least  likely,  keeping 
in  mind  the  prior  odds  (posterior  odds  resulting  from 

the  last  message)  and  the  likelihood  ratios  for  tlie  message. 

Most  likely  AOA:  _ 

Second  most  likely t  _ 

Third  most  likely s _ 

Least  likely  AOA:  _ 

2.  Indicate  how  many  times  as  likely  the  most  likely  AOA 
is  as  compared  with  each  of  the  other  AOAs: 

(a)  Most  versus  second  most  likely  AOA  _ 

(b)  Most  versus  third  most  likely  AOA  _ 

(c)  Most  versus  least  likely  AOA  _ 


appendix  n 

GENERAL  questionnaire  AND 
SUMMARY  REPORT  FORM  USED  IN  THE  RESEARCH  STUDY 


Participant  Number 


Message  No. 


GENERAL  QUESTIONS 


1.  As  the  Division  02^  would  you  request  an  immediate 

briefing  with  the  Division  Commander  after  this  message? 

Yes  or  No  (Circle  One) 

Explain  briefly:  ________________________________________ 


2.  As  the  Division  G2,  would  you  request  additional 
information  from  Corps  after  this  message? 

Yes  or  No  (Circle  one) 

Explain  briefly  _ 


3.  At  this  point,  would  you  conclude  that  the  enemy  has 
selected  a  primary  avenue  of  approach? 

Yes  or  No  (Circle  one) 

If  Yes,  which  avenue  of  approach?  _ 

Explain  briefly  _ 
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Participant  Number 


Date 


SUMMARY  REPORT 


Write  your  intelligence  estimate  at  this  time.  That  is, 
rank  order  the  AOAs  from  moat  to  least  likely  and  justify 
your  answer.  Be  sure  to  include  all  Information  the  G2  will 
need  in  presenting  your  estimate  to  the  commander. 


